Over the past few weeks, there have been relatively few updates about the status of the Depsgraph Refactor project, apart from the regular weekly updates (which have been admittedly a bit vague). That's because I've been busy actually hacking away at it! There's only so much time in a day (or rather, after sleep and research work commitments), so regular blog post updates have taken a bit of a back seat for a while. Nevertheless, today I'll be presenting a little overview of what's been going on, and where we're going now.

Components - The "key" to the new design

Summary of Progress So Far

The WIP state of the codebase can be found at https://github.com/Aligorith/DepsgraphRefactor

This is currently a separate module which may still be subject to “radical flux”. As we'll see shortly, that's exactly what happened earlier this week, which is also kind of why it's being developed this way. Hence, in its current state, it isn't really of much interest to users just yet, though for other coders, it may be a bit enlightening to see what's coming up.

As a bit of a side note, I've used practically the same workflow (i.e. developing refactors outside of Blender's codebase until the core concepts/code stabilises out so that it doesn't require any more major design/architecture revisions) on several prior projects, including Animato system in 2009, Constraints system, Keyframing/KeyingSets stuff, and Action Editor internals refactor 2007/8. Perhaps the only major difference is that this time I have the benefit of a working/usable version control system I can use locally (i.e. in the past, I would have had to work on all of this with no version control until I got the entire thing completely working enough to commit it to SVN). his approach provides a bit more freedom in terms of being able to change direction easily + quickly, but also working roughly in the beginning (i.e. no need to totally satisfy compilers, so can have incomplete stuff + placeholders lying around), and most critically, avoiding accumulating version-control merging backlog (on the off chance that someone modifies the file(s) you've been completely destroying, but also for all the rest of the stuff you aren't touching). There are of course risks involved in this approach, though IMO the base we have is too broken for incremental iterations over the top for that in this case...

The past 3 weeks have roughly amounted to:

GSoC Week 4 = Core API's and Datatypes. That week was basically spent trying to figure out what the base node type (“DepsNode”, upon which all of the other node-types for representing whatever we need) would look like.
GSoC Week 5 = Nodes TypeInfo, ID Node/Groups. From memory, that week was mainly spent defining the “Node Typeinfo” system used to develop specialised node types for representing the stuff we need. On top of that, I started defining the ID Node and ID Group node types (as described in the Overview Design Doc)
GSoC Week 6 = Refined Core API's and Start of Graph Building. As the core system started to look like it was taking shape, I started trying to hack up the graph building code. To make this work though, it soon became apparent that some changes were needed with the core API, which helped make everything fit together better. (For example, I quickly figured out why exactly the call the add nodes happened to be called “dag_get_node()” - that's because not all nodes are fully defined before we try to create the relationships between them, so stubs often need to be added in the meantime, which will then be filled out later when the node is encountered for real. Oops!)

A Change of Direction

By the end of those 3 weeks, the core system was starting to take shape. However, there were still a few unsolved issues (such as how exactly we'd tell each operation callback what it was to work on – making these fully general callbacks was perhaps going to be trickier than initally planned).

More critically though, when trying to figure out how to implement the code to build the dependency graph from scene data, I found that it was quite difficult to find a good way to represent the fact that one node used “the geometry result” of another object, or that another node needed the “final transform” of another object but maybe not necessarily the geometry results. The problem was that with the implementation at the time, you'd have to figure out what ID + PointerRNA was needed to identify the structs involved.

Clearly, the original design wasn't going to work quite as well as initially planned. So, last weekend (or rather, late on Saturday evening), I started mulling over some alternative designs which could work better. By Sunday morning, I already had a preliminary idea of what could be done (just before heading out the door). Since it was a very sunny day, I ended up having a nice long walk outside around the city centre (or rather, what is left of it) with my camera, which proved to be quite effective in terms of being able to think things over. By Sunday evening, I had a firm idea of where we needed to go, and from there, it took just about 2 days to rip out the old node type defines, and replace them with the new ones. What's more, the core framework I'd set up held up perfectly well under these changes, with only minor tweaks needed to get it working with the new setup. If nothing else, we at least know now that if this new design sucks too, we can easily throw it out the window without too much trouble ;)

Design Overview - Version 2

The following diagrams aim to outline how the new design works. There's also a PDF version of these diagrams available from the Wiki: http://wiki.blender.org/index.php/File:DepsgraphRefactor-DesignOverview-02.pdf