What Mercurial Can't Do: Subtree Repos

On the Mercurial website, at the very bottom of this page is a section titled "What Mercurial can't do" which says it can't host related projects together in one repository. Then it links out to the Forest Extension which makes nesting repositories easier to work with (and which comes with TortoiseHg, you just have to turn it on).

Its true that Mercurial can't do this, not even with the Forest Extension. Apparently there are plans for Mercurial to build some kind of subtree repo support in the near future. However, I'm not sure what it will look like. It might not enable the behavior I'm going to discuss here. We'll have to wait and see.

To describe what this really means, here's a possible layout you might find in TFS:

$\
MathUtils\
GraphUtils\
ProjectA\
ProjectA\
MathUtils\
GraphUtils\

In this example MathUtils and GraphUtils are seperate projects which have been written to be reused by other projects. ProjectA is reusing both of them.

ProjectA could just reference $\MathUtils and $\GraphUtils directly. But if there are many projects reusing them and making changes to them that could get hairy. Every time someone in ProjectB changes MathUtils and checks in, they could potentially break ProjectA immediately. To avoid this, ProjectA wants to isolate itself from changes other people might be making to the shared utils. To do this, they create a branch of MathUtils and GraphUtils in their own project. Now they can decide when the right time to bring in changes from other people is, or when the right time to share their own changes with other people is.

To bring in changes from outside in TFS you'd right click on $\MathUtils and say merge, selecting $\ProjectA\MathUtils as the target. To push changes back in TFS you'd right click on $\ProjectA\MathUtils and say merge, selecting $\MathUtils as the target.

In TFS, if you made changes to ProjectA and MathUtils and checked in, you'd get a single checkin containing all changes. When you then merged MathUtils back, the changeset would basically be split so only the changes to $\ProjectA\MathUtils got merged to $\MathUtils (For more on how TFS does merges, check out this post What Mercurial Can't Do: Merge by Changeset). This works because TFS doesn't respect "changesets" across branches.

This does not work in Mercurial. You could setup the same directory tree structure, but you couldn't have just one repository. Instead, you'd have to have $\ProjectA\MathUtils and $\ProjectA\GraphUtils setup as their own repositories nested in the $\ProjectA repository. Mercurial is smart, and it wont try to check in MathUtils or GraphUtils to ProjectA because it recoginizes them to be distinct repos.

So this setup actually works just fine, but it's more work. In TFS, we could do one checkin. In Mercurial, we'll have to checkin changes to each repo individually. That's probably not such a big deal, but in TFS you didn't have to keep track of where you were making changes. In Mercurial you might forget you changed something in MathUtils, and then you might forget to check it in. This is because when you do an "hg status" you'll only see changes in ProjectA. This is where the Forest Extension comes in handy. It adds an "hg fstatus" command which is basically a recursive status command. That way you wont lose track of your changes, but you still have to commit them individually.

But lets think about this for a minute. Is it really a smart thing to do, having one changeset with changes that affect both ProjectA and MathUtils? What's your checkin comment likely to be? Probably something like "ProjectA's xxx feature now can whizbang." If you're looking at the change history for $\Project\MathUtils and you see "ProjectA's xxx feature now can whizbang" does that makes any sense? Does it tell you what changed in MathUtils?

So once again, this is something that Mercurial just can't do. But maybe that's a good thing.

What Mercurial Can't Do: Merge by Changeset

UPDATE 12/1/2011: Mercurial CAN do this!  You used to need to enable the transplant extension.  But since v2.0 it's included out of the box in the form of the graft command.  I should also note that the problems I mention with this approach in TFS do not exist in the Mercurial.  So, basically, go use the graft command and don't read this old post (unless you want to read about how bad TFS is).


Mercurial is a distributed source control system that works very well on Windows and has great windows shell integration with TortoiseHg.

TFS is a centralized behemoth that does source control but also integrates (usually poorly) with every product Microsoft has ever released (not including Bob).

I've used both of these, but I've used TFS much more extensively. I recently started looking into what it would take to switch from TFS to Mercurial and was rather surprised to find a couple things that TFS can do that Mercurial cannot.

The first of these is the ability to do a merge by changeset. In TFS, say you create some branches as follows:

1. Create a new TFS project called Project
2. Check in some source at $\Project\Source
3. Branch that source to $\Project\NewBranch
4. Do 3 checkins to $\Project\NewBranch

Suppose you're not completely done with whatever it is you're working on so you can't merge all three changesets back to $\Project\Source just yet. But lets say your second changeset includes a bug fix to a file that you wont be changing as part of your "main" work in the other changesets. Maybe you want to go ahead and merge this bugfix back to $\Project\Source right now.

In TFS you could do this very easily.

1. In Source Control Explorer, go to $\Project\NewBranch
2. Right click and select "merge"
3. Change to the "Merge by selected changeset" radio button
4. Make sure the target is $\Project\Source
5. Click next
6. On the next page, select only the second changeset
7. Click next and the merge is performed

Ta-Da! You just merged changeset #2 without merging changeset #1 or #3.

You cannot do this in Mercurial, at least, not with a "merge" operation. The only way to accomplish the same type of thing would be to create a patch out of NewBranch and apply it to Source using the hg export and hg import commands.

So the big question is, why can't you do this in Mercurial? The answer goes to the heart of what makes Mercurial so different from TFS. The first thing to realize is that Mercurial does not have "branch lines."

In TFS when you branch code TFS knows that the one is a parent of the other and you can only merge across that branch line. This means you can't do merges between two siblings. For example, in B -> A <- C, you can't merge B to C. You can only merge along the branch lines (Unless you do a baseless merge, which doesn't really count). In Mercurial, the normal way of creating a branch is to simply clone the repository, which means you have a full copy of the entire history of the repo. The image to the right shows what an hg pull would look like when bringing changes from a cloned repository into the original repository.

"A" represents the starting point. "B" represents the first change from the repository we're pulling in. What you can see here is that when you pull changes from mercurial, a temporary "branch" is created containing all the changesets from NewProject in parallel with any changesets from Source.

"C" is the only actual merge, in the tranditional way we think of merging, because it actually brings all the changes together. This is beautiful in its simplicity because until you get to C you don't have to do any work. Each changeset represents what changed from the parent, so you just import all the changesets and associate them with the correct parent. Then only at the very end do you have to do any merges.

A merge in TFS is not so clever. Basically all TFS does is figure out every file that changed on either side, and do a 3-way merge on each in turn, resulting in a new changeset. The upside of this is we can select a single changeset, ignore all the changesets around it, and do a merge.

In Mercurial, you can't pick just one changeset and merge. You have to merge all the changesets before it too because that's the definition of how a merge works in Mercurial. The upside of this is that all the changesets are preserved.

For example, say you want to know who added a certain file. In mercurial, you'll be able to figure this out regardless of what "branch" (cloned repository) it was added in. In TFS, you're screwed because the file will be added in a "merge" changeset. The merge may not have been done by the same person who added the file (in fact, it usually wont be), so to find out who added it, you have to manually follow the branches and inspect the history on each in turn. The same is true (and worse, actually) if you want to know who updated a line in a certain file.

Sadly for me, we're constantly "de-tangling" our changes by doing merges by changeset. But lets think about that for a minute. Is merging a single changeset even a sane thing to do? It turns out, not so much, because its possible for this to result in a broken state. Here's how:

1. Joe Bob adds a new file "hippo.cs" and updates the C# project file
2. Joe Smith adds a different new file "giraffe.cs" and updates the C# project file
3. Joe Smith merges his changeset and ONLY his changeset up
4. The result of the merge does not compile. The error is, "Can not find file "hippo.cs"

WTF!? Why is it looking for hippo.cs?! We didn't merge that changeset! How does it even know about hippo.cs? Its because Joe Bob and Joe Smith both changed the C# project file. When Joe Smith changed it, Joe Bob's changes were already in it. So Joe Smith's final C# project file includes Joe Bob's "hippo.cs" file. But when Joe Smith did a merge, he didn't include Joe Bob's changeset, so the hippo.cs file didn't get merged. And now you're broken.

This happens like all the freaking time with project files (which are the bane of branches and merges). But fortunately it's easy to fix. Just remove the missing files from the project file. But I think you can probably see that if this happened to any file other than the project file, like a real source file, you'd be in a world of hurt.

I'm actually STUNNED, given how much effort TFS puts into protecting the users from themselves that it allows you to merge selected changesets in this way! But it does. And its a feature that Mercurial just can't match, even if it is a feature that can lead to trouble. But maybe that's a good thing.

Merging with TFS

We celebrated "TFS Sucks" day at my office last Friday, so this blog post is a little late.

I don't understand this behavior. I'm merging one branch back to another by selected changeset (so I that I can go through what's changed before I merge). I tell it to do the merge and it comes back with the "Resolve Conflicts" dialog. There was one file in the changeset, so I see one file.

According to the dialog: "Conflicting changes have been detected. To resolve conflicts, select items and click Resolve."

When I right click on it and Compare -> Source to Target... I don't see changes on both sides, I only see the changes I knew I was bringing in.

When I right click on it and Compare -> Source to Base... I see exactly the same changes.

When I right click on it and Compare -> Target to Base... my tool tells me there are no changes.

So, this means there are no changes in my "Target." If that's the case, what "Conflicting changes" have been detected? There is no conflict! Of course, when I tell it Auto Merge All, it succeeds, but why did I have to go through this step at all?

I'm using SourceGear's DiffMerge as my merge tool, maybe that has something to do with it?

Branching, with TFS

Branching is a powerful source control practice that allows you to create copies of the same source code which can be developed in parallel, then merged one way or the other. This allows you to do all kinds of powerful stuff, like:

1. Track releases
2. Isolate different development efforts
3. Maintain stable code

In principle, it's very simple. In reality it can get really complicated. Like, really.

The main difficulty is that what branching strategy you use depends on what you're trying to accomplish. And you may be trying to accomplish different things at different times.

Microsoft has a good guidance document on branching (and TFS) up on codeplex: http://www.codeplex.com/TFSGuide

What I'm trying to accomplish with branches is:

1. Allow developers to develop rapidly
2. Allow developers to promote code which is "done" and prepare it for testing and release
3. Track releases

Promotion Branching
Here's one branching structure that seems like it might accomplish that:

Dev -> Main -> Releases

In this structure, everyone develops in a common Dev branch. When their features are done, they merge them to Main. Then a Release branch is created from which RC builds and the final Release build are made.

It turns out that there is a huge problem with this structure. You have many developers working in the shared Dev branch. If they don't all finish their features at the same time (which they never will...), trouble strikes.

For example, imagine Bob makes some changes and adds a new file to the Project. When he checks into Dev his changeset will include a new file and a modification to the Project file. Now Susy makes some changes and adds a new file to the Project. Her changeset will include a new file and a modification to the Project file as well.

Lets say Susy finishes her work before Bob. That's cause Bob spent all his time on the phone with who knows who and got little to no work done, while Susy only stopped occasionally to check her makeup.

When Susy merges her work to Main by selecting only her changesets, she'll add her new file and her Project file modifications. Her Project file modifications include the file she added, but they also include Bob's file. This is because Bob added his file first. Unfortunately, her merge will not include Bob's new file because she didn't select Bob's changeset.

The result is a broken project file which is expecting to see Bob's new file, but it isn't there. This problem will occur anytime two people make changes to the same file and one of them merges before the other. Imagine how hairy this will be if the file they both changed is a code file and not just the Project file! I get shivers just thinking about it.

So we can't allow many developers to mix "unfinished" code because there is no simple way to untangle it.

Shelving
If we can't let people mix changes, then we can't have a shared Dev branch. Instead, we'll have to have people work completely separate from each other until they're done. We could do this with private branches. Or we can use Shelves.

In TFS, a Shelf allows you to save some pending changes to the server. You can "unshelve" them later, or even share them with other people. This is very similar to having many private named branches, but there is no change tracking.

The problem here is that the shelves are just enumerated in a huge list. So the more projects you have, the more shelves you have, the more of a nightmare working with shelves will be. Also, you have no change tracking.

Why not use private branches? For one, they wont be private, so they'll clutter the crap out of your Source Control Explorer. And for another, everyone says you shouldn't. Including me. No one every says why you shouldn't, including me, but that's what they say. So you should trust them. And also trust me.

By Release
Another approach would be to create our Release branches much earlier in the process. Then we'd work directly in the Release branch. This way, changes that are meant to go out at different times don't get all tangled up.

The problem here is that I don't think you're omniscient enough to make this work. What if a feature takes longer to develop than you thought (cause Bob wont get off the freaking phone). Do you push back the whole release? Do you just try to disable that code? When do you re-enable it?

What if you have many concurrent releases planned, one for next week, and one for the following week? To get the changes from next week's branch into the following week's branch you have to merge through Main. Which is ok, but will be something of a pain.

Plus, you have to make sure that all changes made in a Release branch get merged back into Main.

Clearly, a Release branch is not a good place to do development.

Feature Branching
Yet another approach involves creating Feature Branches. Each Feature Branch isolates the development of a particular feature from the development of everything else. When the feature is done, you merge the Feature Branch into Main and delete it.

If what you're working on is quick and doesn't deserve a whole branch, then just use shelves.

The first problem with this approach is that you have to be pretty omniscient as well. If you start work in a shelf, then realize you'd like to have a branch, the only way to make that work is by using the tfpt unshelve /migrate power tool. At least there's a way, and it's good enough for me, personally, but a lot of people will find it too much (like Bob, who when he gets off the phone is scared to death of the command line).

Another problem is the potential for a proliferation of feature branches. This depends on the size of your project, but you can easily imagine a situation where there are 4 or 5 different features being developed concurrently. And you'll constantly be creating new branches and deleting old ones. That's a lot of branches, and a lot of maintenance.

The Solution
Boy I wish I knew. I think the solution is Mercurial (which really needs a proper website...). But we use TFS. So the best I can come up with is Shelves and Feature Branches. You wouldn't happen to have any ideas I've overlooked, would you?

Vim TFS Integration

This is the second post in my series on using Vim to do C# development. You can read the introduction into why someone might want to do that here.

Where I work, we use Microsoft's Team Foundation Server Source Control. It's not very good at merging, but other than that it's a good tool.

Unlike some other source control tools, in TFS a "get" downloads the latest version of a file, a "checkout" lets you edit a file. Basically all a checkout does is make the file not readonly, but you have to do it or else TFS wont let you check the file in when you're done.

Because of this, if you're using Vim to do your work, when you first open a file and try to go into edit mode Vim will warn you that the file is readonly. This reminds you that you need to check it out. It would be very annoying if you had to switch over to the command line (or VS!) to check it out just so you could start editing. So don't do that! Just add this to your vimrc:

" setup TFS integration
function! Tfcheckout()
exe '!tf checkout "' expand('%:p') '"'
endfunction
command! Tfcheckout :call Tfcheckout()

function! Tfcheckin()
exe '!tf checkin "' expand('%:p') '"'
endfunction
command! Tfcheckin :call Tfcheckin()

The "expand" part will expand to the current file with it's full path included (Note: you have to use the execute command instead of running the ! command directly or else you'll have problems with parenthesis not being properly escaped in your file path).

Now you can simply type :Tfcheckout, and the file will be checked out. When you're all done type :Tfcheckin and the TFS checkin dialog will open, allowing you to enter a comment, checkin any other files, associate the checkin with a TFS work item, etc.

TFS Source Control

The source control system that comes with TFS is pretty good. It's absolutely fantastic when compared with Visual Source Safe (what was so Visual about it anyway?).

It has "Path Space Branching," meaning you can create branches by creating new folders. It follows the Checkout, Edit, Merge pattern by default, meaning many developers can work on the same file at the same time (no locks). It integrates beautifully with Visual Studio, and all of its dialogs and wizards are really straight forward and intuitive.

That said, it is not without its issues...

Anyone who has ever tried to organize a branching strategy for an office of developers knows that branches add complexity, and complexity adds confusion, and confusion slows stuff down. On the other hand, the isolation that branches provide can make life much easier for individuals and individual teams as well as the whole office. This is mainly because you can focus on developing new stuff and not worry about breaking other people or integrating you changes until you're done. Sounds minor, but its a huge help.

However, once you're using branches in this way, you're making the majority of your changes on the "development" branches and then merging back to the "public" branches when you want to release your changes. Enter TFS issue #1.

Merges result in a single changeset, they do not reproduce all the changesets made on the source branch on the target branch.

This means, if you have 5 changesets on your dev branch, with associated checkin comments, and work item associations, and user identification, NONE of that information will appear on the target branch. So when you're trying to figure out:

1. Who made this change?
2. When did this change?
   
3. What has changed recently?

you. are. screwed.

In my opinion, this is so severe it almost makes branches in TFS completely worthless.

But wait, there's more! TFS has branches, and TFS integrates really well with Visual Studio, but its not as perfect as it seems...

Using branches can screw up your project and solution files

It all looks like it should work well. You've got your references set up as project references. You branch everything and the relative paths are all setup the same (..\..\myproject). And yet, somehow VS still gets confused. You start getting errors where the version numbers arn't right. Or it wants to check out all your project and solution files. It's terribly annoying.

And what happens if the relative paths arn't the same? In the branching environment we've setup this happens in some scenarios because branch our project's shared dependencies into the same directory as the main project to isolate the project from all outside changes. When you do this, you have to completely rebuild your project references and solution files because VS gets confused. This isn't the end of the world, but it is a huge annoyance.

It also means, any time you perform a merge, you have to merge the project files by hand because VS will think there are changes (the relative paths) even though there aren't any real changes.

Ready for the next issue?

There is no simple way to rollback a changeset.

The TFS power tools comes with a rollback command, but it's command line only and it's not terribly easy to use (since you need to know the changeset numbers). The absence of this feature in the GUI proper just makes no sense. This is one of those features people always use to sell why you should be using a Version Control system, and TFS doesn't even have it.

I should stop and mention that these are the downsides. The rest of the Version Control that I've used is really great. Workspaces are cool, though only moderately useful. And Shelves are a decent concept, though again, only moderately useful.

The funny part about branching is that when you learn about distributed source control systems, your first reaction is, that would never work, and your second reaction is, that's perfect! Many of the problems you were trying to solve with branching just go away and get replaced with simple communication problems. As in, you have to tell other people when they should pull from you and that kind of thing. However, there are no distributed systems that I currently know of that integrate with Visual Studio like TFS's source control does, nor that are nearly as user friendly. However, I would still encourage you to check them out as it will give you a whole new perspective on versioning.