Render layering with specific depth hierarchy

Nelfie · December 30, 2018, 3:00pm

I’m trying to produce a specific render layering effect. It’s best described by an example:

Let’s say I have three objects:

a ‘set’, in this case a square room with some other geometry in it.
a cube
a torus

The cube and torus are positioned so that the cube is inside the hole of the torus, and part of the torus is spatially behind the cube, and other parts in front.

And I want them to render so that:

the torus is always in front of the set, but equal to the cube.
the cube is equal to the set, and equal to the cube

The end result should look like this: (These screenshots were photoshopped to illustrate the wanted result.)

What I’d like to know is, can I achieve this effect, and if yes, how? If there are several ways, which one would be the most efficient?

I admit, part of the difficulty here is probably my own ignorance on how rendering works in Panda3D. I’ve spent a while experimenting with display region sorting, stencil attributes, projecting renders as textures and so on, but I couldn’t make enough heads or tails of it to figure it out.

If you think you have an idea on how to do it, but need to test it first, here is a file where everything should be more or less set up as needed:

render_layering_template.zip (9.4 KB)

I’m trying to produce a specific render layering effect. It’s best described by an example:

Let’s say I have three objects:

a ‘set’, in this case a square room with some other geometry in it.
a cube
a torus

The cube and torus are positioned so that the cube is inside the hole of the torus, and part of the torus is spatially behind the cube, and other parts in front.

And I want them to render so that:

the torus is always in front of the set, but equal to the cube.
the cube is equal to the set, and equal to the cube

The end result should look like this: (These screenshots were photoshopped to illustrate the wanted result.)

What I’d like to know is, can I achieve this effect, and if yes, how? If there are several ways, which one would be the most efficient?

I admit, part of the difficulty here is probably my own ignorance on how rendering works in Panda3D. I’ve spent a while experimenting with display region sorting, stencil attributes, projecting renders as textures and so on, but I couldn’t make enough heads or tails of it to figure it out.

If you think you have an idea on how to do it, but need to test it first, here is a file where everything should be more or less set up as needed:

render_layering_template.zip (9.4 KB)

rdb · December 30, 2018, 3:26pm

It is easy if you don’t need the torus to depth-test against itself. If you can ensure that the torus polygons will never self-occlude (eg. if the polygons are in back-to-front order and it rotates with the camera, or you split the torus into the inner and the outer part and render the inside first, or you are never looking at it from an angle where it would occlude itself), then you can do this:

Render the set first, normally.
Render the torus with depth test off, but with depth write on.

torus.setAttrib(DepthTestAttrib.make(RenderAttrib.M_always))

Render the cube third, normally.

You can use bins (ie. setBin) to force a specific order, either by putting them in separate bins, or by putting them in a fixed bin and using the second parameter for the draw order. See this page for details:
https://www.panda3d.org/manual/index.php/How_to_Control_Render_Order

Other methods include involving clearing the depth buffer and drawing some objects twice into the depth buffer, but this requires using multiple DisplayRegions and is therefore harder to set up. If the above method will not work for you, it would be easier to turn to StencilAttrib which will provide much greater control over this; you could let the torus write something to the stencil buffer, and draw the set later, telling it to discard any fragments that fail the stencil test.

Nelfie · December 31, 2018, 10:11am

Thank you for your swift reply!

Your first set of instructions would probably not solve the problem, because the torus is not supposed to move or rotate with the camera, so various angles and self-occlusion will have to be accounted for.

Nonetheless, for exercice’s sake, I tried to translate that into code and insert it into the project above. Is this a correct interpretation?

 def renderinglayering(self):
    set = self.set
    torus = self.torus
    cube = self.cube

    set.setBin('fixed', 1)

    torus.setDepthTest(False)
    torus.setDepthWrite(True)
    torus.setAttrib(DepthTestAttrib.make(RenderAttrib.M_always))
    torus.setBin('fixed', 2)

    cube.setBin('fixed', 3)

That results in this:

Is that the correct and intended result?

I also tried splitting the torus into inside and outside parts, but the results was more or less the same.

rdb · December 31, 2018, 10:58am

Only the setAttrib call is needed, not the setDepthTest or setDepthWrite calls, but otherwise, that’s the basic idea.

You don’t need to rotate it along with the camera if you do my trick of splitting it into an inner half and an outer half, like so:

Then, you use setBin to make sure the inner side is rendered before the outer one. I think this should cover the self-occlusion cases, as long as you are not rendering it double-sided.

Nelfie · January 1, 2019, 1:41pm

Ok, I tried to correct the code. It should look like this, correct?

def renderinglayering(self):
    set = self.set
    torus = self.torus
    cube = self.cube

    set.setBin('fixed', 1)

    torus.setAttrib(DepthTestAttrib.make(RenderAttrib.M_always))
    torus.setBin('fixed', 2)

    cube.setBin('fixed', 3)

Here’s the result:

I also tested a version with the torus divided as you showed, I think:

The code looks like this:

 def renderinglayering(self):
    set = self.set
    cube = self.cube
    torus_inside = self.torus_inside
    torus_outside = self.torus_outside

    set.setBin('fixed', 1)

    torus_inside.setAttrib(DepthTestAttrib.make(RenderAttrib.M_always))
    torus_inside.setBin('fixed', 2)
    torus_outside.setAttrib(DepthTestAttrib.make(RenderAttrib.M_always))
    torus_outside.setBin('fixed', 3)

    cube.setBin('fixed', 4)

The result was this:

In both cases, the cube renders behind the torus instead of inside of it; so I’m probably doing something wrong. Is there maybe custom prc setting that might affect it? I’m using a modified ‘Config.prc’, rather than the default.

rdb · January 1, 2019, 10:09pm

Have you put them back in the same display region? The cull bin order only works within each display region, so you cannot control rendering order between different display regions with it.

You can use display regions to control the order if you wish, but it’s a bit more code. You would just need to make sure that the display regions follow the same order as we had specified with setBin, and that there is no depth clear set on any of the subsequent display regions, and that the cameras have the same lens.

serega-kkz · January 1, 2019, 10:37pm

This is not possible, the cube will either be completely behind or completely up. Of course, if you make a ring cut (Torus) in the right place. However, this will only work in 2D, since the camera cannot be moved. Of course, you can always make a ring facing the camera in two parts, but we don’t know what you want in the end.

Nelfie · January 2, 2019, 10:48am

That’s the answer! I did indeed have each object in a different display region, so of course .setBin() wouldn’t work.

I reworked the split torus version with all objects in base.render, and it renders as you described:

I also tested it with a rotating camera and that works too.

Now, some outstanding issues are the obvious lighting difficulties for the torus object (a seen in the artifacts in the shadows), and that a more complex object than a simple torus like this would probably harder to split.
(Of course, I acknowledge that I never cleary said that I might also need this rendering trick for objects other than toruses.)

In the hope to solve that, I tried your suggested method using display regions. The code should look something like this, yes?

 def renderinglayering(self):
    display_region_set = self.display_region_set
    display_region_torus = self.display_region_torus
    display_region_cube = self.display_region_cube

    lens_set = self.camLens

    # I'm using base.camLens because I simply reparented base.camera 
    # to use it for rendering the set.

    camera_cube = self.camNode_cube
    camera_torus = self.camNode_torus

    camera_cube.setLens(0,lens_set)
    camera_torus.setLens(0,lens_set)

    display_region_set.setSort(0)
    display_region_torus.setSort(1)
    display_region_torus.setClearDepthActive(False)
    display_region_cube.setSort(2)
    display_region_cube.setClearDepthActive(False)

I probably managed to sneak in some mistake somewhere again, because it didn’t work:

@serega-kkz, as the first screenshot in this post testifies, rdb’s method using .setBin actually works. But thank you for your concern!

rdb · January 2, 2019, 11:22am

Do you still have the depth test set to M_always when rendering the torus? You still need to do that for this trick to work, but doing that will bring up the self-occlusion problem again, so in that sense using display regions doesn’t really help you directly.

If you want the torus to render with correct depth testing to handle the self-occlusion problem more generally, you might try this method instead, which does require ordering using at least 2 display regions because it involves clearing the depth buffer:

Render the set
Render the cube
Clear the depth buffer
Render the cube to the depth buffer only, using setAttrib(ColorWriteAttrib.make(ColorWriteAttrib.C_off))
Render the torus

Maybe this would work? It makes sure that the cube will be rendered depth-tested against the set, and that the torus will be rendered depth-tested against the cube but not the set, and everything is still depth-testing against itself properly.

Alternatively, you could try the stencil buffer (which doesn’t need multiple display regions), maybe something like this:

Enable stencil buffer using framebuffer-stencil true in Config.prc and stencil-bits 1
Render the torus first, with a StencilAttrib that writes a 1 to the stencil buffer.
Render the set, with a StencilAttrib that discards the pixel if the stencil buffer contains 0.
Render the cube.

serega-kkz · January 2, 2019, 1:56pm

I had no doubt that .setBin works. However, this does not solve your problem I was talking about it.
You need to divide everything into two parts. Rendering the back of the very first in line, and the front last.

Nelfie · January 3, 2019, 4:52pm

RECAP for everyone else:
I tried the first solution offered in rdb’s last post, and with some further assistance from him over the Panda3D IRC channel, we managed to get it to work. I’ll provide an detailed explanation and example of how to do it once the thread comes to a close.

@rdb
It only came to me a while after I left the IRC channel, but I wanted to ask: Would the second solution you propose, which is to use the stencil buffer, be more efficient than the first one, performance-wise?

As you describe it, it would require only one copy of each object, while the current one needs a second instance of the cube, along with the second display region. This is fine for the example project, but if one were to replace the cube with a more complex object, wouldn’t performance be proportionally impacted? Or do you think this would be about even with using a stencil buffer?

If you think using the stencil buffer would or could be more efficient, I’d like to try that out too before bringing the thread to an end.

@serega-kkz
Oh, I see! My apologies.
But there’s good news: the first solution proposed by rdb in his previous post actually works, giving the exact result shown in the doctored screenshot.

rdb · January 3, 2019, 4:56pm

You do have to render the cube twice, but I doubt that it’s going to have a measurable impact on performance (as long as it you don’t replace that cube with a complex scene consisting of many independent objects). I don’t know if using a stencil buffer adds a performance penalty. I suggest you should do whatever works or is easiest for you until you find measurable performance issues.

serega-kkz · January 3, 2019, 5:05pm

I would like to see the result in the form of a screenshot of what was required. I think I do not quite understand you in the first message.

Nelfie · January 4, 2019, 9:04am

@rdb
I see. Thank you for your insight!

I’ll write a summary of the first solution then, along with an example file. I thought I’d do it today, but forgot it was going to be a bit too busy. I think I’ll manage within the next days, however.

@serega-kkz
I’ll provide a detailed explanation of the solution and an example file as soon as possible, but the intended result is really nothing more or less than what is shown in the two latter screenshots of the original post:

I wanted to find a way to render three groups of objects as follows:

a 1st group (the set), which renders as normal. (i.e. ‘normal’ meaning that objects within that group render on top of each other and occlude themselves and each other in the usual, spatial back-to-front way.)
a 2nd group (the torus), which will always render in front of the 1st group regardless of the relative depth between the 1st and 2nd group, but renders normally against itself and the 3rd group.
a 3rd group (the cube), which renders as normal, both in relationship to itself, as well as the 1st and 2nd group.

Nelfie · January 6, 2019, 9:02pm

BASIC RENDER LAYERING AND MASKING EFFECT

Here is a way to produce the effect described in the original post. The concept of it was explained by @rdb in this post.

WHAT IS THE GOAL?

We want to produce the illusion that some models (the ‘overlay’, or ‘overlay models’; e.g. the green torus in this screenshot) will always render in front of everything else (the red room and pillars), except for one or more select models (the dark blue cube in the center).

These select models will occlude or be occluded, fully or partially, by the overlay, according to their depth. In other words, the overlay and the select models render against each other as normal.

This produces an interesting effect when the select models are occluded by non-select models: the parts of the overlay that would be occluded by the select models will be transparent, producing a cut-out effect in the shape of the select model.

ONE WAY TO DO IT

The technique is simple in theory: To produce the illusion that only some select models can occlude the overlay, we create duplicates of these select models and set them to be transparent, but still be capable of occluding the overlay. The duplicates are necessary because the overlay is rendered in a separate display region.

In practice, it requires a number of steps.

To make this work, we first separate the overlay and the duplicates from everything else. Depending on your intentions, and how you wish the overlay models to interact with the rest, you might have to figure out a specific hierarchy.

For the sake of this example, we separate each group into its own ‘scene’ (i.e. an orphan NodePath to which they are directly or indirectly parented, and containing its own camera. ‘base.render’ would be an example of a scene, as provided by default)

We now have:

‘main scene’, containing select and non-select models. It is rendered into a first display region by it’s own camera
‘overlay scene’, containing the overlay models and the duplicates. It is rendered into a second display region, by a second camera.

After creating the camera, pass the pos and hpr values of the first camera to the second camera, so that they have identical position and orientation. This should also be true for select objects and their duplicates, so if you have moved the select objects after creating their duplicates with “.copyTo()”, you will also have to pass on their pos and hpr to the duplicates.
This is done via object_b.setPos(parent_b, object_a.getPos(parent_a)); the same goes with ‘.getHpr’ and ‘.setHpr’.

If the cameras and objects are supposed to move during runtime, you can leave the paragraph above out for now, and write a continuous task that will pass on pos and hpr from the first camera to the second, and from select objects to duplicates every tick.

You also have to ensure both camera lenses have the same settings. This can be achieved by getting the lens node of the first camera, and callin ‘.setLens(lens)’ on the second camera.

After this, we might first need to sort our display regions. The overlay display region needs to have a higher sort value than the main display region. The default of 0 would work due to a kink of how Panda3D operates, but for the sake of clarity and preventing unexpected issues later, it is best to change it. Set the sort value with ‘.setSort()’.

Second, we set the overlay display region to ‘.setClearDepthActive(True)’. This ensures that Panda3D will clear all data from the main scene still in the depth buffer before rendering the overlay. Doing this might not be necessary if a previous display region (‘previous’ in terms of of sorting) already has that setting and uses a different part of the window than your overlay does.

Thirdly, we need to place the models in the overlay scene into the ‘fixed’ bin and give them appropriate sort values. You need to give the duplicates of the select models a higher sort value than the overlay models. This ensures the duplicates will write their depth data into the buffer before the overlay models are rendered. That way, the overlay models can be fully or partially occluded by the duplicates. To do this use setBin(‘fixed’, v), where ‘v’ is your sort value, for each object.

Finally, we need to give all the duplicates a rendering attribute which tells Panda3D to render them transparently. To do this, pass .setAttrib(ColorWriteAttrib.make(ColorWriteAttrib.C_off)) onto each duplicate model. With this attribute, the objects’ colours will not be rendered, but they will still provide depth data so that the overlay models can be occluded by them.

And now, the desired effect should be achieved. If not, review the steps and see whether you have not skipped anything. You can also compare your code with the example file below.

Render_Layering_Example.zip (13.5 KB)