Saturday, November 13, 2010

Bullet SoC - Doing the SVN Dance - Part 1

In order to start work on this project again, especially on Point Cache support, I figured it was time to grab a more up-to-date Blender trunk to work from, since many things have changed there AFAIK, not to mention in a few other places too.

The "Traditional" Approach
Those people who've heard one or two things about SVN (the revision control system we use for the codebase) or even the developers of it, would say "Why don't you just do a merge to HEAD?"

If you fall into this category, please read this before continuing. Basically, SVN's "merge" capabilities ("log" capabilities are equally bad, especially in TortoiseSVN's GUI) are both sluggish (especially on a stressed server located half-way around the world, though even a server several blocks away on campus is still quite bad) + fickle to work with (it's very easy to get conflicts or missing changes from specifying the wrong bounds for the merging).

It doesn't help when the server's SVN version is several years old (IIRC, it's something like 1.4.x or worse, which means that the "improved" merge-tracking stuff isn't available) or that the version of TortoiseSVN I updated to a few months ago is even more defunct than older versions in this regard (put simply, it refuses to do anything at all when clicking through the merge dialogs, even after trying every last combination of options).

Furthermore, to compound this misery, consider that the branch that you need to merge changes from trunk into was created about over half a year ago, and hasn't had a single merge from trunk in that time (given the defunctness of the TortoiseSVN version involved, and the desire to limit downtime appeasing the svn gods).

The Turnaround
Having worked a bit with command-line SVN now (thanks to becoming the de-facto SVN guru a SE group project at uni), I finally felt comfortable enough with using "vanilla" SVN to install such a client on my computer.

Among the benefits this brings is that now I can finally delete the old SoC branch that I never used (stupid (Tortoise)SVN refused to overwrite the existing branch when trying to branch trunk for the current branch, but TortoiseSVN doesn't offer any way of deleting branches that don't exist on your HD!). Another is that now I also get the revision numbers in my splash screens (I was too lazy to install commandline SVN for JUST that purpose when Tortoise already covered my standard SVN needs!)

A Modern Twist
While thinking through my options for how to get an updated branch based off trunk, I decided that it's probably less work rebranching from trunk and then reapplying all changes to this new branch, given that most of the changes were fairly localised.

However, the problem of keeping this up to date was still unresolved.

What I really wanted was a way to just update my working copy "normally" and have the updates reflected, without doing any of the stupid merging dances. In essence, this is really what using a branch should really have been all about IMO. Any other crap about keeping track of "ancestry" really ends up compromising the basic cases.

Enter my latest idea: why don't I try and set up my working copy so that it can be updated to/from 2 different svn branches at a time? That is, doing an update from trunk works perfectly normal, but I can still commit these and other changes to my branch while I keep developing this stuff, and later on I could just commit all these changes to trunk normally too!

After a previous "incident" while trying to update Blender's copy of Bullet, I realised the branch that SVN will commit to/from depends only on what is stored inside its little private ".svn" folders. If we could just trick it into using other little private folders like this, or just replacing the contents of these directories, then maybe we have a hope of achieving this goal.

After some investigations, it looks like it might actually be possible! In particular, it helps that I'm working on Windows, as there's "Official support for Windows '_svn' directories" since SVN 1.3, which is enabled by defining an environment variable "SVN_ASP_DOT_NET_HACK". If one branch's metadata is stored in ".svn", the other's could be stored in "_svn". Defining this environment variable per (commandline) session would hopefully be enough to force SVN to only be able to "see" one of these sets of directories, which in turn would allow one branch at a time to be used.

Well, at least that's my theory, based on reading up about this option.

When I have some more time, I'll go and try to test this out for real - probably I'll need to hack up a few scripts to make this easier to do. And hopefully if this works, then my SVN branch-management problems will at last be solved... at least until I move to Linux, though by that time, we might finally have moved to a distributed VCS like git (which apparently handles merging much better - merging then will only depend on CPU speed vs Network Latency + Server Speed). Fingers crossed...


  1. I can't tell you how happy I would be if blender moved to git, or even hg for that matter.

    It is definitely something I plan to campaign for once 2.5 gets out of beta.

  2. May I suggest you to use Mercurial or Git for your local copy ?

    With mercurial (and hg-subversion), the workflow may be like this:

    a) checkout (clone) the blender svn repository as a hg/git repository
    b) do what you want in your local repository, even lot of commits
    c) update from the svn repository. It may create branches. Merge them when you want.
    d) continue working

    When you are done and ready to push onto the svn trunk, rebase your commits on top the svn trunk, commit, and push.

    I use this solution to work on blender addons and it's really nice. The only issue is that you need first to clone the blender svn repository, which can takes some weeks ;) (Hg need a local copy of every commits, and ask them one by one to SVN, which is really slow).

    But you can avoid fetching all the revisions with --startrev.

    Examples: (with HG)

    # CLONE
    $ hg clone --startrev HEAD
    destination directory: blender
    [r33047] nexyon: Missed that in last commit.
    pulled 1 revisions
    updating to branch default
    4113 files updated, 0 files merged, 0 files removed, 0 files unresolved

    # Now, I can work in my repository and do some "local commits"

    $ cd blender/
    $ echo "dog" >> Makefile
    $ echo "dog" >> Makefile
    $ hg diff
    diff -r 1478d8c5cb04 Makefile
    --- a/Makefile Sat Nov 13 13:47:33 2010 +0000
    +++ b/Makefile Sat Nov 13 18:28:31 2010 +0100
    @@ -58,3 +58,5 @@
    $(MAKE) -C $@ $@ || exit 1;


    # LOCAL commit
    $ hg commit -m "add two dog at the end of the makefile, ouaf"

    $ hg log
    changeset: 1:200312faa23c
    tag: tip
    user: Guillaume <....>
    date: Sat Nov 13 18:30:08 2010 +0100
    summary: add two dog at the end of the makefile, ouaf

    changeset: 0:1478d8c5cb04
    user: nexyon@954f8c5b-7b00-dc11-b283-0030488c597c
    date: Sat Nov 13 13:47:33 2010 +0000
    summary: Missed that in last commit.

    # Is there something in trunk that i missed ?
    $ hg incoming
    incoming changes from

    # Unfortunatly, there is not.

    # Check what can we send
    $ hg out
    comparing with
    changeset: 1:200312faa23c
    tag: tip
    user: Guillaume Bouchard
    date: Sat Nov 13 18:30:08 2010 +0100
    summary: add two dog at the end of the makefile, ouaf

    # There is only one changset

    At the end, you can use hg push to push your local commits back to the trunk.

    The only issue remaining is "how to get sync with the trunk"

    a) "hg pull" to get new commits from trunk
    b) If there is new commits, a new local branch will be created, you can do two things :

    1) rebase your changes on top of it. (i.e. saying that the parent of your commit is the svn HEAD).
    2) merge you branch with the svn HEAD

    Both options are a bit different. I invite you to read the doc or ask a question, I'll be more than happy to help you.

    Good luck !

  3. Ok, I lost my long comment because of really nice comment system (or because I'm stupid and I don't keep a copy before sending it)

    In short: use git or mercurial on your computer with their addons to work with svn. You'll then handle your merge from your side, with more atomics commit, and stuff may be easier.

    If you wanted, I can take some times to show you how. I'm on irc on approximately every blender channels on freenode with nickname guibou.

    Good luck and do not let the tool annoy you !

  4. Guillaume:
    It seems your original comment got spammed. Restored that now.

    Part of the reason that I'm currently not considering this approach that you suggest (in fact, I did consider doing it) is that the work is not going straight into svn trunk, but rather into another svn branch.

    If I'd just use git/hg to do a local branch for this work, there'd still be distribution problems (pbo only has svn support, otherwise I'd have to use some other 3rd party server :/). Also, from the sounds of things, I'd still only be able to commit to trunk.

    Still, this will still be my fallback route if the experiment fails :)


    Should be able to do some git-svn magic to make merging between trunk and your branch super easy (using git's merge tools) and then pushing to your upstream svn branch from within git.

    Too bad you already deleted the bullet branch on the svn server because you probably could've done a (relative) pain-free merge with trunk and then kept going.