Design Process

How do you design software?

Is that topic too wide for a single blog post? It certainly is, but I'm going to focus on some of the big pieces, and ignore lots of rabbit holes.

First things first, I'm talking about the kind of software that I spend my 8-5 working on. Namely, business type software with lots of data to capture, a fair amount of process, and a number of different "classifications" of user. This process will not apply to Operating System development, or multi-media application development, or sales websites, etc.

With that out of the way, here are my steps:

1. Investigate the process this software is intended to solve
2. Find out what data needs to be captured
3. Design the UI
4. Design the data queries and updates
   
5. Design the database
6. Design the code structure
   

UI First
I believe this fits firmly in the "UI First" category. Of course, on my list UI is #3, but #1 and #2 are really just reminding you to do good analysis of the problem before you start designing.

UI First is a pretty widely accepted approach. Here's some evidence:

• Rick Schaut: UI Design
• Jeff Atwood: UI-First Software Design
• Jeff Atwood: The User Interface Is The Application
• Matz on Craftmanship
• 37 Signals: Getting Real, Step 1: No Functional Spec
• Joel on Software: Painless Functional Specifications

If the last two seem to be contradicting each other, go read what Joel means when he talks about Functional Specs. His specs are all built around and focused on the UI, and how it will behave, including mockups of the UI. So, I count it as UI-first, even if he may not have completely intended it that way.

BDD
I'm not too up on today's newest fad: BDD. But, I think my steps align fairly well with the goals of BDD, though they certainly aren't the same. BDD supporters will probably complain that I put "design the database" before "design the code structure". I would argue these are more or less interchangeable, but that database before code makes more practical sense for a number of reasons. For one thing, doing the db first allows you to use the Active Record pattern instead of the Repository pattern, if that makes more sense for your app. It also allows you to use a wider array of tools (Linq to SQL, Entity Framework, Ruby on Rails (scaffold!), etc).

Data First
Not database first, data first. There is a difference!

Its easy to take all the "entities" you know you will need and lay them all out into database tables, normalizing as you go. But databases can represent the same thing in a surprising number of different ways. So if you're just looking at the entities and data elements and laying out a database from that you're shooting in the dark.

You need to figure out how you're going to be using (querying, updating) this data first. For example, what searches will you be doing on what data elements?

This may lead you to discover that you will always return data from two entities together. If you'd blindly created two tables which contain many of the same columns, you'd constantly be unioning them. This will not only be a coding problem but could even become a performance issue, depending on how complicated things get. Maybe it would be better to de-normalized a bit and put both entities in the same table.

De-normalizing is not the only change you may make when you think about queries and updates first. As a real example, I once had a situation where I decided to add a bunch of bit columns to a table. These columns would represent if certain attributes where present or not. I was going to need to return entities that had some of those attributes on or off based on the person who was logged in. With bit columns my query would require mapping from the person's attributes to parameters on a stored procedure to the bit columns on the entity. It would also require lots of chained "where ( @bit1 is null or @bit1 = tbl.bit1 )" statements. This would all be a real pain to update when a new bit arrived, not to mention its a lot of repetitive code.

But since I thought about the query first, I realized I could represent the bits as rows in a child table of the entity. If the bit was on, there was a record for it in the child table, if not there was no record. Then the query simply became an inner join dramatically reducing the amount of code needed and making the whole thing more maintainable.

Don't take this too far. I'm not suggesting you prematurely optimize everything. But the database is there to support your application, not dictate how it should behave. So it should be designed to support the application as best it can.

How to design the UI
I suspect the reason why UI design isn't done first more often is simply because its a real pain. The simplest things can be ridiculously difficult. Sometimes you struggle with screen real estate. Other times you struggle with representing relationships (ex: parent, child).

But coming up with a UI concept is not nearly as hard as prototyping your UI. Laying out a UI, in Visual Studio lets say, can be a real challenge. Sure, for simple UIs its a breeze, but in my experience most UIs don't stay simple for long. Because the UIs are so hard to layout, there is real pressure not to change them. Please just find the first thing that looks like its probably passable and lets call it a day!

This simply comes down to a tooling problem. Visual Studio is not a great UI prototyping tool. Even if you avoid the slippery slope to perfectionville and remember your working on a prototype there's a problem. The problem is, it looks too good! When you show it to people it looks so similar to what the final product will look like, they wont be able to get past the fact that it's not perfect. They'll constantly be on you about "Why is that not lined up right?", "Why doesn't this look better", "Couldn't you have made this not suck so bad?"

What about whiteboards? Its easy and fast to sketch up some UI on a whiteboard. But, there are two problems here.

1. Scale. Its much too easy to cheat on a whiteboard. You can make things fit that will never fit or make things look good that really wont. And you wont find out until you actually go to put together the UI...
2. Unprofessional. If you want to share this with a client, or do a "screen demo" you'll be showing a bunch of digital photos of chickenscratch on a whiteboard. I know Peopleware says unprofessional is a word used by weak frightened managers, but in this case the truth is no one is going to be impressed by your whiteboard pictures. They might even think you're just lazy.
   

Then there are paper prototypes. There are whole books written about paper prototypes and how to do them. They are more complicated then simply drawing a picture on paper with a straight edge so it doesn't look too crappy. The problem with that is when you want to tweak anything you'll probably have to completely redraw. So paper prototypes use note cards and post it notes and pieces of cut out paper and glue sticks and tape. That way you can swap pieces in and out and make more changes without redrawing everything.

But, to me this seems like a lot of work. I want to prototype a UI, not enjoy craft time with my scissors and straight edge and glue sticks and note cards and post its and different sizes of paper and digital camera...

That's why I like Balsamiq Mockups. Its kind of like Flash meets Visio, so its flexible and very easy to use. I've been using it for only a week or so but I'm really enjoying it. It solves the "too perfect" problem by using images that look hand drawn. So when you show it to people, they can focus on what matters: how the user will inteact with the interface. And unlike my white board sketches, I find the look to be quite appealing.

To see what it looks like, check out their samples. This is a best of both worlds type of situation. Its easier to build and change than paper, faster to save snapshots, AND it comes out looking surprisingly professional, without looking so similar people can't focus.

And now you code
Of course, coding is the easy part... Ha! But, its easier when you have a good design. And hopefully because you have a good design, you'll encounter less re-work and fewer surprises.

Getting Stuff Done

Today I was reminiscing about college and doing a bit of compare and contrast between how I got stuff done then and how I get stuff done now. I was surprised to find that this was actually somewhat interesting, so I'm writing about it.

In college, each class gives you one assignment at a time, and each comes with a due date. You may have many things going on concurrently, but each is clearly prioritized based on how hard you know the assignment will be, and when that assignment is due.

This situation is quite manageable. You may have to pull some late nights to get your stuff done on time when you find yourself in the unfortunate position of having many assignments due all at once. You may also shoot yourself in the foot by procrastinating. But when that happens, you know it was your own fault. So, at least in my experience, college assignments were really quite doable. I had some periods of stress, and I certainly had times when I felt very busy, but overall, it wasn't so bad.

The real world is not like that. But, why not? What makes it so different?

Here's some of the characteristics of college that don't show up so frequently in the real world:

1. Well defined work
2. Fixed schedules
3. Clear priorities
   
4. No re-work
   
5. No support calls
   
6. No releases

This is pretty self explanatory I think, but I like to hear myself type, so, lets talk about it!

Well defined work is easy. In college, you get a detailed assignment. It tells you what your inputs are, what your outputs are, exactly how the programs should behave in every way. You write it, test it, and you're done! Well defined work. In the real world you try to get your customers to define the work. But your customers don't even know the work themselves, so its really hard for them to define it for you. This means you can spend lots of time working on things that you either don't need, or that don't work quite the way they really should. And you don't get to find this out until you're all done.

Fixed schedules is also easy. In college, you get two weeks to do your assignment, from start to finish and turn it in. The professor never tells you to stop a week in and do another project. Or worse, and more realistically, to add another project on top of the first. This seems to happen all the time with real work, and it makes it difficult to plan ahead and stay on track. You start to feel like a never ending stack trace...

Clear Priorities dove tails nicely with fixed schedules. Your school assignments are easy to prioritize because you know up front how much of your grade its worth, how difficult its going to be, and how long you have to do it. These factors allow you to easily decide which assignment should be given priority. Once again, real life is not so clear cut. Things pile up and they're all ASAP highest priority do or die holy crap! Good luck prioritizing that!

Ah yes, re-work... Here's a factor that you never ever ever ever ever encounter in college. If you suck it up on an assignment you get an F, you move on. Not so in life beyond the academic gates. If you screw it up it doesn't end there. It snow balls and things get worse and worse. And only if you're lucky do you actually get the opportunity to go back and fix what you screwed up. And now you've spent twice as long on that assignment, which has certainly pushed back some other absolutely top highest priority assignment.

And now we come to support calls. My favorite! I never had a professor call me weeks after I'd finished an assignment and ask me how to use it. Or ask me why it doesn't do something no one ever heard about before. But the best part about support calls is how much time they can eat up. Simple calls will take you about twice as long to answer as you feel like it actually took. And complex ones will eat your entire day.

Finally, with a college assignment you have a single release: the first one. And none after that. In the real world, you'll have lots of releases. How many depends on your specific situation. But, one of the tenets of Agile is release often, so I think lots of people are probably doing lots of releases these days. Releases take time. You have to migrate the database, release the client, push the update, test. And the bigger the system, or the more components it has, the harder this process will be. In fact, this can get ridiculously hard. And this is yet another thing you never had to deal with in school.

So, is it possible to get stuff done in an environment where you're being pulled in so many directions at once? Or is the only answer to bring some of that collegiate structure back?

Interestingly, if you read many of the Agile practices in this light, you see a lot of it is geared towards restoring some of these characteristics. For example, iterations. This practice manages to restore some of your well defined work, fixed schedules, and clear priorities. Also, the customer representative and release often practices are there to try to cut down on the amount of re-work.

So maybe the best you can do is try to bring back as much collegiate structure as you can, because its awfully hard to get anything done without it.

Web Apps Are Better

I am a Windows Application developer. My company also writes Web Applications when necessary, but by and large, we do Windows Applications.

So you can imagine how distressing it is for me to admit that Web Apps are better. I've written about the topic of Web UIs vs Rich Client UIs in the past, but didn't quite commit to Web Apps in general being better. This time, I'm gonna go into it a bit more.

There are two primary reasons that people started writing Web Apps in the first place:

1. Easy access from anywhere
2. No installation means no system administration

The funny thing about these two reasons is they arn't really completely correct anymore. You can use Web Services to create "smart" clients that can access central data but still run anywhere. And Click Once allows you to install an app on even low privilege computers easily.

As for installation, any non-trivial app is going to require all kinds of ActiveX plugins, addins, and whizbangs to allow people to do anything more complicated than type in text boxes (ie. upload/download files, etc). These things are arguably more of a nightmare to maintain than a Windows App since they have a way of working one minute and not the next.

Plus, of course, you have to deal with browser incompatibility.

But, all that said Web Applications are still better. Don't worry, I wont make you wait any longer to find out why:

It's because they're prettier and more usable.

That they are prettier is not a huge surprise. HTML/CSS makes it much easier to do cool layouts incorporating all sorts of graphics. Also, for some reason, its simply expected these days. If you made a Web App that looked like a Windows App, everyone would immediately think it was crap. For some reason, there is just a higher standard for appearance on the web.

You may at first question if being pretty really makes it better. But the first time you show a client an immensely complicated Windows App, with all sorts of exciting features (both from a business stand point and a technology standpoint) and then you show them a simple website, and they go "ooh! ahh!" over the website, you'll start to wonder, maybe there is something to this pretty stuff after all?

After the 5th time it happens, you'll stop wondering and starting being pissed off about it...

That they are more usable is a huge surprise. At least to me. Applications on the web are so clearly inhibited by the Web environment, you would think it would be impossible for them to be more usable. Unless you have the resources of Google at your disposal. And everything is still in "Beta."

First off, they are slow. Constantly contacting a remote server for every little thing... The Window keeps going white while the next page is fetched from the server. AJAX makes the experience significantly better, but is still slow. Fortunately, internet speeds got faster. And now, especially with Chrome, browsers are getting faster too. And best of all, users don't really seem to mind! As long as the net connection isn't so slow that every server trip takes 30 seconds, people seem perfectly happy to wait for the page to save. I think it's simply because they are used to it, and they expect it to be that way. Plus, developers know they are slow. So we write Web Apps with that in mind by rendering less at one time, and sending less to the server. Basically, we simplify things. And in the end, this often leads to a more usable application.

Next, they are inconsistent. There are lots of emerging standards, but there is no HTML tab control. No "navigation" control. No "grid" control. So everybody does there own thing. And everybody who ends up choosing a similar metaphor, like tabs, implements them their own way.

Interestingly, this doesn't seem to really matter. In fact, I would argue its a main part of why Web Apps are more usable. I've talked about consistency before and argued that consistency for its own sake is a bad thing. If you have a reason to do something different, then do it. As long as you are consistent within your own App, you're good. The web forces this mentality on your designs. Each time, you start from scratch. And so you end up designing something completely based around the specific requirements of that app. Not based on the old out dated UI metaphors that everyone else uses. And even when what you design looks like one of those old out dated metaphors, it has to be custom implemented, so it may not behave exactly the same way. Instead, it behaves in whatever way you've determined is best for your app. And as far as I can tell, this seems to be good enough.

Even the most computer illiterate people aren't so dense that they can't adapt to a slight difference. In fact, from what I've seen, the more illiterate they are, the less likely they are to notice it was different at all! Within reason, obviously.

The last thing is that web apps are harder to write. Well, at least, they used to be. I don't think this is true anymore either. The technology for creating Web Apps has evolved at a striking pace. Especially in the last few years. 5 years ago "classic" ASP was one of the most modern Web tools. Now you have Ruby on Rails and ASP.NET MVC. AJAX. JQuery. These are great tools. And they are making it possible to do very complicated things very easily.

Ridiculously easy, in some cases. I have seen things done on the web in no time which took forever to do in a Windows App. And, they were arguably done better. And I'm not talking about layout issues either, which have always been easier on the Web than, say, Windows Forms.

Speaking of Windows Forms... A lot of the reason why the Web has been able to catch up and even pass Windows Apps is due to Win32. Win32 sucks. Try to create a suggest field in Windows Forms using no 3rd party components, I dare you. You'll be at it for months, and it still wont be quite right. Meanwhile, your web developer friend will do it in an afternoon.

I think WPF fixes nearly all of this. It makes it so you can actually build controls and UIs, instead of hacking them together. It makes me think its possible that Windows Apps could catch up. If only the text wasn't blurry, and everything didn't go out of focus when you scrolled, and lines that were 1 pixel always displayed... (All of these are problems in WPF due to Microsoft's insistence that WPF be dpi independent. A noble goal. But a dumb one, if it means you can't do anything with text without all your users saying, "why is this blurry?")

From a very high level, these are some of the things that should have made Web Applications worse, but seem to have actually made them better. Honestly, I think its because the Web has built in constraints. You have to make round trips to the server, and render a new page every time. That's why the web lends itself so naturally to MVC. Windows Apps don't. You also have to recognize that you can't send too much data or performance will suffer. Its easier to ignore this on Windows Apps, until its too late. Also, you can't display popups, or modal dialogs, or keep tons of stuff in memory.

So all the downsides of the Web turned it to plus sides. Lucky. But perhaps we should look at some specifics of what makes Web Apps more usable.

Tabs vs. Scrolling
In Windows, you see that most applications pick a minimum size, and they try to make everything fit within that area. So if the minimum supported screen size is 1024x768, you'll see that all the data entry fields (text boxes, combo boxes, grids, etc) are all cramped into that space. If they don't fit, then tabs are added, so that you can switch between them.

On the web, you just scroll. Usually horizontal scrolling is avoided, but no one minds vertical scrolling. Turns out this really simplifies things. For example, what is going to be saved when you click the save button. Its obvious, everything on the page. What about with the tab control. Will it save everything in all the tabs, even though you can't see it? Do you have to save each tab individually? Is what you typed on the tab even going to be there after you switch to another tab and then come back? Everyone knows how tab controls work right? So none of this matters. But still, you can see there is much more potential for fear with the tabs than with the scrolling.

Menus vs. Navigation
In Windows, you get "File menus." "File", "Edit", "View", etc. This is how you navigate around to the things you need to work on. For some reason, the menu titles are always made fairly generic (like "Format" in Word). So all kinds of stuff that is related in function but unrelated in purpose get grouped together.

On the Web, you get Navigation. And it comes in all kinds of forms. Sometimes the navigation looks like tabs! Other times it looks like menus, though the names are always by purpose instead of by function... Other times its just a list of links down the left of the page. Or, sometimes the navigation is only available from the home page, and you return there whenever you finish something and need to start something else. All of these models make it easy to support many levels of navigation, which is essential on the Web, since you need smaller surfaces for performance.

Its interesting to look at the modern push in application development. Look at IE7, where'd the menu bar go? Look at Windows Media Player. Look at Google Chrome. Look at Word and Excel with the ribbon! Or look at the Zune software. These are all very "web" feeling to me. Even the ribbon, in some ways, feels more like the kind of toolbar you would find on a web site.

Wizards vs. Pages
In Windows there is a separation between Wizards and "Forms" or "Screens" or whatever you wanna call them. A Wizard always pops up and lets you step through something. A Screen is just there and lets you move around, enter data, save stuff, type stuff, whatever you might be doing.

On the Web, there is no such thing as a Wizard... When you need to step through stuff, you just have pages that step you through. You use the browser's back button to go back, or the web site provides a back button for you on the page. Sometimes the Next button says "Next", other times it says "Continue to step 2" and other times it says "Enter Account Info ->" Its simple because its really no different any other page from a user's standpoint.

Popups vs. Popups
In Windows Apps you've got popus all over the place.
"YOU JUST DID A SEARCH THAT RETURNED NO RESULTS"
"THERE IS A VALIDATION ERROR SOMEWHERE ON THIS PAGE, GOOD LUCK FINDING IT!"
"I HOPE YOU NOTICED THAT THERE IS SOMETHING IMPORTANT ON THIS PAGE THAT YOU SHOULD PAY ATTENTION TO, GOOD LUCK FINDING IT!"

Web apps tend to display popups for "Are you sure you want to delete this?" and that's about it (Other than Rickrolls...). Instead the web will display messages throughout the page when neccessary. This is clearly much friendlier.

Modal Dialogs vs. Modal Dialogs
You find lots of Modal stuff in Windows too. Which is really great, cause in Windows when a modal dialog is open, you can't resize, minimize, or move the window the dialog is displayed over. So not only can you not do two things at once in that application, you can't easily do two things at once in Windows either!

Modal Dialogs have started to appear on the web too. Typically they're a div that opens over the rest of the content, and the stuff in the back gets grayed out to focus your attention. Its a neat trick, when used rarely and for important stuff only. Since its harder to do than Modal Dialogs in Windows, I'm hoping it will stay rare, though I've already seen some good examples of misuse. Especially by some newer "feedback" sites that some web apps integrate with.

MDI vs. umm, nothing?
This is my favorite one. In Windows apps you often find lots of windows. MDI was the first way of handling this, where within your main application window you would get many other windows. But this sucked, so now adays you tend to see tabs instead of windows. This is better.

But the web doesn't do this. At all. You've got one window. If you want another one, you open another one. Now you have two. But they are not related to each other, they don't really know about each other. You're basically just running two instances of the application.

Because of this, Web applications focus much more on what I'll call flow. Moving you from one place to the next. That's why the back button is so important and generally useful. Yes, its harder to look at two things at once (but that's true in MDI and tabs too) and its harder to switch between two things too. But, oddly, this seems to be not such a big deal. And when you want to do it, just open two windows!

Grids vs. Flow Layout
In Windows Apps, especially heavily data oriented apps, you see LOTS of grids, and lists. They're everywhere, and they're horrifically ugly, and terrible to use. Apart from the fact that it is really nice to click on a column header and, bang!, change how the grid is sorted, grids don't add much value. There just there because the tooling makes it really hard to repeat stuff over and over without them.

On the web, you see lots of what I call Flow Layouts. They're grids, kind of. But, they're pretty. They're custom formatted. Titles are bold, some text is colored, some isn't. Font sizes change based on the importance of the text. Look at Amazon.com's search results for example. There are no column headers at all! To sort, you just choose some options from a dropdown at the top.

Now, I'm no usability expert. But I don't think you have to be to realize that the way Apps work on the web is just better than the way they work in "Rich Clients". And this is completely ridiculous. The Web can't even come close to matching the capability and flexibility of a rich client. And yet, the Web is still better. Hopefully WPF will help, since you can actually create Web-like displays with it that don't perform horrifically. But what really needs to happen is people need to realize that just because you're making a Windows Application doesn't mean you need tab controls and menu bars and properties forms and grids and modal dialogs. You can use a Back button in your Windows app, if you think it will help.

Of course, this is an incredibly difficult mold to break out of. All the tooling is against you. All your corporate "UI" standards are against you. Your own brain is against you. But I encourage you to at least try to incorporate some "web-ish" UIs into your next rich client project. Maybe you can get even get some "oohs and ahhs" of your own?