Wednesday, November 12, 2014

Dev Ramblings - Grease Pencil OpenGL Render Exports

While doing some testing of the Grease Pencil functionality last week, I came across some weird/unexpected behaviour regarding how some of the OpenGL renders were working. As many will know, these are currently the only way to get Grease Pencil drawings out of Blender. And as I found out, there are quite a few annoying quirks that I came across, the most serious of which is shown below:

Blegh! What is up with those regions that are completely transparent, and fringed with light gray?!


Alpha Handling for OpenGL Renders
The glitch above is the cumulative result of two changes made (or rather pushed for) by the "strict technical correctness" (*) color-grading/video-editing crowd...

(*) On an off-topic note: The world of color representation (i.e. color spaces, color coordinate space systems, gammas/mappings/LUT's, illuminants, color perception, etc.) is a bottomless pit of endless depravity... The worst part is that some of the things out there (e.g. the 3+ versions of "Delta-E" aren't even all that entirely accurate for what they for all but a few special use cases). Gah! You have been warned - You probably should know about it all, but you will also wish you didn't know/don't need to know/shouldn't have tried to know... </end rant>

1) If alpha < 1, all chromaticity information is inherently invalid. Therefore, if alpha < 1, you get to have any colour under the sun, provided it's a shade of gray. This is at least half the bug, as this change now affects any image buffer in Blender where there are partial alpha values anywhere, which happens to be on just about every bloody Grease Pencil stroke, because that's the way we usually use them, and that's the only way we can implement them to accumulate the way they do to look nice in the normal viewport!

2) Colours need to be in linear colour space for certain processing steps. This wouldn't really be that much of an issue, except that the sequencer uses a special version of this processing, which ends up outputting colours which are consistently 1-2 notches "brighter" (aka whiter, less saturated, and fainter-looking)...


As a result of these glitches, we get the following crappy situation:
What it looks like in the viewport - This is what the end result should be!

This is what happens in the Image Editor preview once we're rendered the image. As can be seen, the areas with partial opacity and with many stroke points layered on top of each other, have been made completely transparent (fading off as the accumulation grows less). Secondly, of the colours which are left, they are all faded to gray. All due to the way that these changes have interacted with the alpha blending stuff used by Grease Pencil drawing in unexpected ways...

And here is the rendered result - The colours are washed out and all blown out, no matter what you do. Clearly something isn't right here!


Potential Fixes:
After a lot of experimentation, the only solution I found was to manually knock out the alpha channel values (i.e. forcibly set all alpha values on the entire OpenGL renders) to only have alpha = 1.0. An alternative may have been to only do a GL_RGB read of the pixels, but then that wouldn't have worked with all the other image render machinery that Blender uses :(

There is another downside here in that by knocking out the alpha values like this, we effectively make it impossible to do certain types of background-replacement effects in the Sequencer with such renders later, since the alpha channel will have been made to say that the whole image is fully opaque.


Sky Background Colors
From a test file from pepeland, I discovered that currently the sky background gradients are NOT drawn in OpenGL playblasts, despite the fact that we can do it perfectly fine in the viewport for realtime usage. There really isn't any excuse here that we only use the bland horizon colour here... It makes things look quite crap, as below:

 Combined color + alpha in the Image Editor - Things are looking quite gray and weird already...

"Just" the color pass in the Image Editor [Actually, that's the saved render] - Things are really starting to get weird now... I don't recall those lines being pink!

As an illustration of the problems alluded to above, here is what's in the "alpha" channel of the render (Darker = More Transparent). This apparently leads to some weirdness in how the colours start getting handled...

Finally, this is what it's supposed to look like - As seen from the standard viewport.


Closing Words
At this stage, I'm not sure what the best way to patch our way out of this mess is going to be...

Option 1: Fix the sky backdrop issue by itself, then add an option to the OpenGL Playblast operator to nuke the alpha channel (aka set all alpha values to 1.0) when reading out the screen buffer (and prior to saving).

This would likely be the least controversial of all the approaches, though it would be less user friendly from a Grease Pencil for art/final output perspective, as then we have to find ways of exposing the new option.


Option 2: Create a new dedicated operator for rendering out Grease Pencil stuff, all OpenGL powered, for which we can have full control over which quirks we support/don't support.

This would be better for artists in having a tool optimised for these use cases, which still uses the same infrastructure used for viewport drawing (i.e. same engine, same processing). But, code-wise, we'd end up with a duplicate codebase for the rendering output part (namely, a near complete copy of render_opengl.c)


Option 3: We forget about OpenGL output as being the solution for output of such shots from Blender, and concentrate instead on getting GPencil Strokes -> Freestyle Strokes working, allowing us to have higher quality line drawing + the nicer output the renderers can do for the other bits and pieces.

This one we will still do eventually, but whether it's so easy to accomplish without delaying the release of the rest of this functionality is another matter...

12 comments:

  1. Great post algorith! How about grease pencil strokes to image paint strokes?

    ReplyDelete
  2. Hmm... I haven't considered that :)

    Something you may not know is that very first prototype was in fact a solution where we just had a screen-aligned image that you can draw on. Not as useful as being able to draw in 3d too, and it suffered from resolution problems (i.e. you can't easily resize the viewport and keep things without pixellation) along with some other issues I can't remember right now.

    Back on topic: When you say "convert to image paint strokes", what exactly are you referring to? Converting Grease Pencil strokes to image paint when done in the Image Editor? Or

    ReplyDelete
    Replies
    1. Maybe something along the lines of what you suggest in your answer to the next post, a sequencer strip where you select grease pencil layer(s) and a render method, maybe a scene camera or just the original 'drawn' perspective, there a lot of possibilities - I still haven't tried a build of this branch as my build skills went out of date with the switch to got but I saw some interesting experiments on vine, 'volumetric' paint being one of them, I'll try and dig out the URL if you like...

      Delete
    2. Actually just solved that - Daniel Lara who made the vines I referred to also uploaded a build for windows to graphicall with links there to the tests he's done...

      Delete
  3. Hi Aligorith!! Sorry for double posting, but I posted maybe a little too late in an older post about the sequencer. Do you think fixing these opengl issues may allow us to preview the grease pencil in the sequencer (the ones in the 3D view of the scene of course)??. Sorry, I'm just dying for this feature and I can't seem to find it in the roadmap. In your older post about "dev roadmap" I talked about why it it is so important and powerful... as well as in bf-commiters. Ups... triple posting then, jaja sorry, but I don't know if you had read them. Amazing work as always...
    David.

    ReplyDelete
  4. Hi David,

    I was looking into this earlier today. I think the main reason the strokes from the 3D view aren't included is because Scene strips in the Sequencer act as placeholders for the rendered scenes. As we know, rendered scenes do not include Grease Pencil strokes.

    There are probably two ways we can do this:
    1) Add a type of sequence strip specially for rendering out Grease Pencil strokes - This would allow us to be able to specially render these out. The problems arise when you have strokes <-> geometry occlusion, but you don't want to use the OpenGL rendered objects.

    2) An option in the scene strip to include these

    3) Otherwise, the freestyle based approach might win out
    2)

    ReplyDelete
  5. Hi Aligorith, thanks for the answer.

    About the possibilities you mention:

    1) Although this might work, it would be more like a workaround. I'd love to have it, better than nothing, but I doubt this is the cleanest way to do it (workflow and ui wise)

    2) I think this would be the best choise, just an option in the scene strip. This way we can keep the beautiful and fast workflow of not having to pre render anything to edit animatic scenes.

    3) This might work, but it would be only for final render, so then again the exceptional non render editing workflow would be missing the powerful grease pencil. This would be more of a solution for later, but not for animatic/layout editing, where the grease pencil shines greatly.

    Another solution that comes to mind, also as a "workaround" for now, would be to improve grease pencil converting abilities, like for example to automaticly make it a curve with radius, and also , if there are keyframes, make more than one curve and convert those keyframes to object visibility. This way we could have animated strokes (as curves with radius) ready for the sequencer or other crazy usages. The downside is that we would lose colors, volume strokes, and all the other amazing stuff you are currently developing.

    Thanks,
    David.

    ReplyDelete
  6. Hi there!
    For me, as user, simply expect press "OpenGL render active viewport" botton and get
    the same image as in the standard viewport, only that, no need to be forced to make a "real" render.

    About Freestyle option/solution , It´s a great great idea indeed, lot of artistic potential here (so excited about that :) , but if it´s the only
    way to get a clear render output, I think it´s more like a "workarround" when you only want to get a simply quick viewport
    capture.

    Maybe there is an another solution-workarround, I'm making some test and I'll post the results

    Daniel

    ReplyDelete
  7. Hi Again

    Maybe the simpiest solution is make a blender screencast (Alt-F3), need some improvements like capture only a region or viewport and capture all the frames, but works realy well, what you see is what you get.

    Maybe a custom "screencast" button in the GP panel?


    http://oi58.tinypic.com/dgkz2p.jpg
    [IMG]http://i58.tinypic.com/dgkz2p.jpg[/IMG]


    Daniel

    ReplyDelete
  8. But wouldn't the screencast option be the same as opengl rendering the active viewport?? You would still need to import pre-rendered images to the sequencer, therefore loosing the non render editing workflow already existent.

    ReplyDelete
  9. Oh sorry, I don´t talk about squencer solution, just try find a easy way to get render output from the viewport because the opengl rendering the active viewport don´t work propelly with grease pencil stuff.

    ReplyDelete
  10. I didn't see any code but
    the render problem may comes from wrong Unmultiplied/Premultiplied Alpha conversion or alpha over.
    as for the opengl render problem, it could be integer overflow of ushort alpha.

    ReplyDelete