Edge outline Cg shader?

I am implementing an edge outline cg shader.
The basic idea is simple. First, I get the depth buffer using a depth shader.
The output color is the depth.

void vshader(float4 vtx_position : POSITION,
uniform float4x4 mat_modelproj,
uniform float4x4 mat_modelview,
out float4 l_position : POSITION,
out float4 l_color0: COLOR0) {
l_position = mul(mat_modelproj, vtx_position);
float depth = mul(mat_modelview, l_position).w*.15;
l_color0 = float4(depth, depth, depth, 1);
}

void fshader(float4 l_color0: COLOR0,
out float4 o_color : COLOR) {
o_color = l_color0;
}

Then, I use another shader to do post-processing.
The edges are extracted and bolded.

void vshader(float4 vtx_position : POSITION,
float2 vtx_texcoord0 : TEXCOORD0,
out float4 l_position : POSITION,
out float2 l_texcoord0 : TEXCOORD0,
uniform float4x4 mat_modelproj)
{
l_position = mul(mat_modelproj, vtx_position);
l_texcoord0 = vtx_texcoord0;
}

void fshader(float2 l_texcoord0 : TEXCOORD0,
uniform sampler2D tex_0 : TEXUNIT0,
uniform float2 windowsize,
out float4 o_color : COLOR,
uniform float4x4 mat_modelproj)
{
float sepx = 1/windowsize.x;
float sepy = 1/windowsize.y;
float4 color0 = tex2D(tex_0, l_texcoord0);
float2 texcoord1 = l_texcoord0+float2(sepx, 0);
float4 color1 = tex2D(tex_0, texcoord1);
float2 texcoord2 = l_texcoord0+float2(0, sepy);
float4 color2 = tex2D(tex_0, texcoord2);
float2 texcoord3 = l_texcoord0+float2(-sepx, 0);
float4 color3 = tex2D(tex_0, texcoord3);
float2 texcoord4 = l_texcoord0+float2(0, -sepy);
float4 color4 = tex2D(tex_0, texcoord4);
o_color = (color0-color1).z > .007||(color0-color2).z > .007||(color0-color3).z > .007||(color0-color4).z > .007?
float4(0, 0, 0, 1) : float4(0, 0, 0, 0);
}

Parameters in the snippets are hard coded for quick test.

My problem is there is always a discontinuity at an arrow tip, as is shown in the following figure.


I also tested adding dark pixels in 8 directions and different texel step sizes, but the results look the same. I would appreciate your help to denote my mistake.

This is a bit of a guess, but it could be that in the case of narrow regions–such as the tips of your arrows–the samples are being taken too far afield, resulting in them “skipping over” the regions in question.

Perhaps this might be improved by taking additional samples, closer to the point being tested?

1 Like

There are probably ways to tweak the algorithm in order to get better results. For one, you could try tweaking the way you are sampling around the fragment, by also sampling diagonally.

2 Likes

Thank you both for your quick reply. It turned out to be the “skipping over” problem. My winsize is 4k, but the getxsize, getysize function returned 800x600.

1 Like

After some further struggling, I got stuck at the window sizes. The texel size of the TEXCOORD is received from setShaderInput at the very beginning. When resizing the window, the shader fails to get updated. Is there a good way to notify the shader about texel size changes?

You could perhaps register a window-event handler, and then call “setShaderInput” within the associated method, I think.

Something like this (untested):

# In your intialisation:
    self.accept( 'window-event', self.windowEventHandler) 

# Elsewhere:
def windowEventHandler(self, window = None):
    self.postProcessNodePath.setShaderInput("windowSize", Vec2(window.getXSize(), window.getYSize()))
2 Likes

You can use uniform float2 sys_windowsize in a Cg shader to automatically get the size in pixels of the window, without needing to set it yourself using setShaderInput.

3 Likes

Thank you very much. The sys_windowsize worked. I also found another way by using the dexeloff parameter provided by tex2D. But my problems remained.

Yesterday I thought it was because of the “skipping over” problem caused by a large step. But today I noticed the arrows are not even shifted. The following figures show an example. I am shifting the depth texture 10 dexels to the left. Vertical arrows seem to be correctly shifted, but the horizonal ones (green) has no shadows.
image

Addition Note 1:
I did more test afterwards. I am shifted the TEXCOORD instead of the dexeloff. The step lengthwas set to ±15*1/sys_windowsize.x in the x direction and ±15*sys_windowsize.y in the y direction. Likewise, the horizontal green arrow failed to get a shadow on its left side. The vertical arrows got shadows in all four directions.
image

Addition Note 2:
The Cartoon filter is correct and the green arrow has a shadow in the left. It must be my problem.
image

Looking at your second-to-last image there, it seems to me that the horizontal arrows do have offset “shadows”.

It’s just that, because of the slope of the arrows–because, I suppose, the eye tends to measure distance at that range by closest distance, not horizontal distance–they look closer than they are.

See here, taken from that screenshot:

arrows

Thanks a lot. I may not accept it to be a visual difference…for the cartoon filter does not have the problem. I am shifting equally in x and y directions.

Looking again at your image, it may be that part of the problem is that you’re not actually getting equal offset in both directions.

Look at one of the vertical arrows: the horizontal “shadows” are offset by ~16px, while the vertical “shadows” are offset by ~29px. So it’s perhaps both a visual difference and an actual one:
arrow

Hmmm, I computing the texcoord using sys_windowsize.x, and sys_windowsize.y, as follows.

float sepx = 1/windowsize.x;
float sepy = 1/windowsize.y;
float4 color0 = tex2D(tex_0, l_texcoord0);
float2 texcoord1 = l_texcoord0+float2(sepx, 0);
float4 color1 = tex2D(tex_0, texcoord1);
float2 texcoord2 = l_texcoord0+float2(0, sepy);
float4 color2 = tex2D(tex_0, texcoord2);
float2 texcoord3 = l_texcoord0+float2(-sepx, 0);
float4 color3 = tex2D(tex_0, texcoord3);
float2 texcoord4 = l_texcoord0+float2(0, -sepy);
float4 color4 = tex2D(tex_0, texcoord4);

My expectation was 1/sys_windowsize.x returns the size of one texel represented in 0~1 scale.

Try disabling power-of-two texture resizing, just to rule this out as a possible cause of error:

textures-power-2 none

Thanks for further follow-up.
Here is the result after disabling non-standard texture sizing.
image

I nearly gave up implementing a new shader. Today, I changed my mind to inherit CommonFilters and implement the outline edge shader by modifying the cartoon ink filter. The original cartoon ink used aux (normals). My plan is to replace it using depth.
After toggling on the depth for the “cartoon ink” lines:

        if ("CartoonInk" in configuration):
            needtex.add("aux")
            needtex.add("depth")
            auxbits |= AuxBitplaneAttrib.ABOAuxNormal
            needtexcoord.add("aux")
            needtexcoord.add("depth")

and editing the first line and commenting out the last line of the program-generated fragment shader, I got

void fshader(
float2 l_texcoord : TEXCOORD0,
uniform sampler2D k_txcolor,
uniform sampler2D k_txaux,
uniform sampler2D k_txdepth,
uniform float4 k_cartoonseparation,
uniform float4 k_cartooncolor,
uniform float4 texpix_txaux,
uniform float4 texpix_txdepth,
out float4 o_color : COLOR)
{
o_color = tex2D(k_txaux, l_texcoord);

float4 cartoondelta = k_cartoonseparation * texpix_txaux.xwyw;
float4 cartoon_c0 = tex2D(k_txaux, l_texcoord + cartoondelta.xy);
float4 cartoon_c1 = tex2D(k_txaux, l_texcoord - cartoondelta.xy);
float4 cartoon_c2 = tex2D(k_txaux, l_texcoord + cartoondelta.wz);
float4 cartoon_c3 = tex2D(k_txaux, l_texcoord - cartoondelta.wz);
float4 cartoon_c4 = tex2D(k_txaux, l_texcoord + cartoondelta.xz);
float4 cartoon_c5 = tex2D(k_txaux, l_texcoord - cartoondelta.xz);
float4 cartoon_c6 = tex2D(k_txaux, l_texcoord + float2(cartoondelta.x, -cartoondelta.z));
float4 cartoon_c7 = tex2D(k_txaux, l_texcoord - float2(cartoondelta.x, -cartoondelta.z));
//float4 cartoon_c8 = tex2D(k_txaux, l_texcoord + 2cartoondelta.xy);
//float4 cartoon_c9 = tex2D(k_txaux, l_texcoord - 2
cartoondelta.xy);
//float4 cartoon_c10 = tex2D(k_txaux, l_texcoord + 2cartoondelta.wz);
//float4 cartoon_c11 = tex2D(k_txaux, l_texcoord - 2
cartoondelta.wz);
//float4 cartoon_mx = max(cartoon_c0, max(cartoon_c1, max(cartoon_c2, max(cartoon_c3, max(cartoon_c4, max(cartoon_c5,
//max(cartoon_c6, max(cartoon_c7, max(cartoon_c8, max(cartoon_c9, max(cartoon_c10, cartoon_c11)))))))))));
//float4 cartoon_mn = min(cartoon_c0, min(cartoon_c1, min(cartoon_c2, min(cartoon_c3, min(cartoon_c4, min(cartoon_c5,
//min(cartoon_c6, min(cartoon_c7, min(cartoon_c8, min(cartoon_c9, min(cartoon_c10, cartoon_c11)))))))))));
float4 cartoon_mx = max(cartoon_c0, max(cartoon_c1, max(cartoon_c2, max(cartoon_c3, max(cartoon_c4, max(cartoon_c5,
max(cartoon_c6, cartoon_c7)))))));
float4 cartoon_mn = min(cartoon_c0, min(cartoon_c1, min(cartoon_c2, min(cartoon_c3, min(cartoon_c4, min(cartoon_c5,
min(cartoon_c6, cartoon_c7)))))));
//float4 cartoon_mx = max(cartoon_c0, max(cartoon_c1, max(cartoon_c2, cartoon_c3)));
//float4 cartoon_mn = min(cartoon_c0, min(cartoon_c1, min(cartoon_c2, cartoon_c3)));
float cartoon_thresh = saturate(dot(cartoon_mx - cartoon_mn, float4(3,3,0,0)) - 0.5);
//o_color = lerp(o_color, k_cartooncolor, cartoon_thresh);
}

The fshader directly draw the aux normal buffer to the screen. Its result is as follows. The far objects are penetrating forward.
image

If I use the original setting (no depth buffer):

    if ("CartoonInk" in configuration):
            needtex.add("aux")
            # needtex.add("depth")
            auxbits |= AuxBitplaneAttrib.ABOAuxNormal
            needtexcoord.add("aux")
            # needtexcoord.add("depth")

I get the correct aux normal buffer (does not seem to be a correct normal buffer though). There is no penetration.

How can I correctly toggling on the depth buffer here?

If I do not change the shader string and run the program directly, the results before and after toggling depth are as follows.

Before

        if ("CartoonInk" in configuration):
            needtex.add("aux")
            auxbits |= AuxBitplaneAttrib.ABOAuxNormal
            needtexcoord.add("aux")

image
After (several penetrations)

        if ("CartoonInk" in configuration):
            needtex.add("aux")
            needtex.add("depth")
            auxbits |= AuxBitplaneAttrib.ABOAuxNormal
            needtexcoord.add("aux")
            needtexcoord.add("depth")

image

If I change the filter to setAmbientOcclusion(), which uses depth texture underneath, the penetration also exists.

float sepx = 1/windowsize.x;
float sepy = 1/windowsize.y;

I suspect that the problem with your shader might be here: Presuming a standard sort of window-resolution, “windowsize.y” will be less than “windowsize.x”. As a result, “1/windowsize.y” will be greater than “1/windowsize.x”, and thus will produce a greater offset.

Looking at our page for CG shader-inputs, I see that we have a uniform named “texpix_x”, which the documentation indicates “[c]ontains the U,V offset of a single pixel in the texture”. This might be a better value for your purposes than one based on the window-size.

(Note that the “x” at the end indicates the name of the texture in question, not the x-axis, I believe.)

Thanks. The tepix_x seems to be automatically generated when a shaderinput texture is set. But I do not have an external shaderinput. I tried get a depth buffer from base.win. However, it does not seem to be accessible to the shader. I went through previous Q&As and successfully rendered the depth buffer to a texturecard, but I could not get the correct value in a cg shader. I also tried inheriting CommonFilters and revising the program-generated code there. Like my previous posts, the textures[“depth”] does not seem to be correct and the textures[“aux”] also does not look like a normal buffer. I search the internet as well as the documents for reasons but did not find a good reading material for them. In the end, I gave up making the depth+normal based outline edge. Instead, I added boundaries to different color sections and implemented a color-based outline edge.

I am going to close this question. My final solution is as follows. I inherited the CommonFilters and added an edge outliner considering colors.

CARTOON_BODY_1 = “”"
float4 cartoondelta_n = k_cartoonseparation * texpix_txaux.xwyw;
float4 cartoon_n_ref = tex2D(k_txcolor, %(texcoord)s);
float4 cartoon_n0 = tex2D(k_txaux, %(texcoord)s + cartoondelta_n.xy);
float4 cartoon_n1 = tex2D(k_txaux, %(texcoord)s - cartoondelta_n.xy);
float4 cartoon_n2 = tex2D(k_txaux, %(texcoord)s + cartoondelta_n.wz);
float4 cartoon_n3 = tex2D(k_txaux, %(texcoord)s - cartoondelta_n.wz);
float4 cartoon_n_mx = max(cartoon_n0, max(cartoon_n1, max(cartoon_n2, cartoon_n3)));
float4 cartoon_n_mn = min(cartoon_n0, min(cartoon_n1, min(cartoon_n2, cartoon_n3)));
//float4 cartoon_n4 = tex2D(k_txaux, %(texcoord)s + cartoondelta_n.xz);
//float4 cartoon_n5 = tex2D(k_txaux, %(texcoord)s - cartoondelta_n.xz);
//float4 cartoon_n6 = tex2D(k_txaux, %(texcoord)s + float2(cartoondelta_n.x, -cartoondelta_n.z));
//float4 cartoon_n7 = tex2D(k_txaux, %(texcoord)s - float2(cartoondelta_n.x, -cartoondelta_n.z));
//float4 cartoon_n_mx = max(cartoon_n0, max(cartoon_n1, max(cartoon_n2, max(cartoon_n3, max(cartoon_n4, max(cartoon_n5,
//max(cartoon_n6, cartoon_n7)))))));
//float4 cartoon_n_mn = min(cartoon_n0, min(cartoon_n1, min(cartoon_n2, min(cartoon_n3, min(cartoon_n4, min(cartoon_n5,
//min(cartoon_n6, cartoon_n7)))))));
float cartoon_n_thresh = saturate(dot(cartoon_n_mx - cartoon_n_mn, float4(3,3,0,0)) - 0.5);
“”"
CARTOON_BODY_2 = “”"
float4 cartoondelta_c = k_cartoonseparation * texpix_txcolor.xwyw;
float4 cartoon_c_ref = tex2D(k_txcolor, %(texcoord)s);
float4 cartoon_c0 = tex2D(k_txcolor, %(texcoord)s + cartoondelta_c.xy);
float4 cartoon_c1 = tex2D(k_txcolor, %(texcoord)s - cartoondelta_c.xy);
float4 cartoon_c2 = tex2D(k_txcolor, %(texcoord)s + cartoondelta_c.wz);
float4 cartoon_c3 = tex2D(k_txcolor, %(texcoord)s - cartoondelta_c.wz);
float4 cartoon_c_mx = max(cartoon_c0, max(cartoon_c1, max(cartoon_c2, cartoon_c3)));
float4 cartoon_c_mn = min(cartoon_c0, min(cartoon_c1, min(cartoon_c2, cartoon_c3)));
//float4 cartoon_c4 = tex2D(k_txcolor, %(texcoord)s + cartoondelta_c.xz);
//float4 cartoon_c5 = tex2D(k_txcolor, %(texcoord)s - cartoondelta_c.xz);
//float4 cartoon_c6 = tex2D(k_txcolor, %(texcoord)s + float2(cartoondelta_c.x, -cartoondelta_c.z));
//float4 cartoon_c7 = tex2D(k_txcolor, %(texcoord)s - float2(cartoondelta_c.x, -cartoondelta_c.z));
//float4 cartoon_c_mx = max(cartoon_c0, max(cartoon_c1, max(cartoon_c2, max(cartoon_c3, max(cartoon_c4, max(cartoon_c5,
//max(cartoon_c6, cartoon_c7)))))));
//float4 cartoon_c_mn = min(cartoon_c0, min(cartoon_c1, min(cartoon_c2, min(cartoon_c3, min(cartoon_c4, min(cartoon_c5,
//min(cartoon_c6, cartoon_c7)))))));
float cartoon_c_thresh = saturate(dot(cartoon_c_mx - cartoon_c_mn, float4(3,3,0,0)) - 0.5);
float cartoon_thresh = saturate(cartoon_n_thresh + cartoon_c_thresh);
o_color = lerp(o_color, k_cartooncolor, cartoon_thresh);
“”"

1 Like