Bug in cartoon shader sample 1.9

On arch linux 64 with a nvidia GTX660 card with drivers 325.15 both the basic and advanced shader results in a white dragon/dino. What additional info can I provide to help fix this bug?

Well, I had the same problem, and I think the “lightingGen.sha” is the problem :
I had to rewrite it, and now it works:
Save yours with another name, and create a new “LightingGen.sha” by copying these lines :

//Cg
//
//Cg profile arbvp1 arbfp1

void vshader(in float2 vtx_texcoord0 : TEXCOORD0,
             out float2 l_texcoord0 : TEXCOORD0,
             float4 vtx_position   : POSITION,
             float3 vtx_normal     : NORMAL,
             float4 vtx_color      : COLOR,
             out float4 l_position : POSITION,
             out float4 l_brite,
             out float4 l_color    : COLOR,
             uniform float4 mspos_light,
             uniform float4x4 mat_modelproj)
{
  l_position = mul(mat_modelproj, vtx_position);
  float3 N = normalize(vtx_normal);
  float3 lightVector = normalize(mspos_light - vtx_position);
  l_brite = max(dot(N,lightVector), 0);
  l_color = vtx_color;
  l_texcoord0 = vtx_texcoord0;
}


void fshader(in float2 l_texcoord0 : TEXCOORD0,
             uniform sampler2D tex_0 : TEXUNIT0,
             float4 l_brite, 
             float4 l_color     : COLOR,
             out float4 o_color : COLOR)
{
  if (l_brite.x<0.5) l_brite=0.8;
  else l_brite=1.2;
  o_color=l_brite * l_color;
  float4 texcolor = tex2D( tex_0, float2( l_texcoord0) );
  o_color = texcolor * l_brite;
}	

You’re welcome :wink:

What exactly did you change?

Well, I don’t know what I changed, because I copied that from an other topic, but my default “LightningGen.sha” was not the same (I saw some differences, like float4 was float3…).

Just try, if it doesn’t work, I don’t know what to do… however, I had the same white dragon problem, and it’s solved now :wink:

Here is the diff. It didn’t work for me. The advanced shader made a black dragon :smiley:

--- lightingGen.sha     2013-04-13 05:03:24.000000000 -0700
+++ lightingGen2.sha    2013-10-07 13:23:54.403350156 -0700
@@ -2,11 +2,13 @@
 //
 //Cg profile arbvp1 arbfp1

-void vshader(float4 vtx_position   : POSITION,
+void vshader(in float2 vtx_texcoord0 : TEXCOORD0,
+             out float2 l_texcoord0 : TEXCOORD0,
+             float4 vtx_position   : POSITION,
              float3 vtx_normal     : NORMAL,
              float4 vtx_color      : COLOR,
              out float4 l_position : POSITION,
-             out float4 l_brite    : TEXCOORD0,
+             out float4 l_brite,
              out float4 l_color    : COLOR,
              uniform float4 mspos_light,
              uniform float4x4 mat_modelproj)
@@ -16,14 +18,19 @@
   float3 lightVector = normalize(mspos_light - vtx_position);
   l_brite = max(dot(N,lightVector), 0);
   l_color = vtx_color;
+  l_texcoord0 = vtx_texcoord0;
 }


-void fshader(float4 l_brite     : TEXCOORD0,
+void fshader(in float2 l_texcoord0 : TEXCOORD0,
+             uniform sampler2D tex_0 : TEXUNIT0,
+             float4 l_brite,
              float4 l_color     : COLOR,
              out float4 o_color : COLOR)
 {
   if (l_brite.x<0.5) l_brite=0.8;
   else l_brite=1.2;
   o_color=l_brite * l_color;
-}
+  float4 texcolor = tex2D( tex_0, float2( l_texcoord0) );
+  o_color = texcolor * l_brite;
+}

Are we sure its the shader thats at fault?

I loaded up panda, and he is shaded nicely for me. Replace the dragon code with this:

        self.character = Actor()
        self.character.loadModel('panda')
        self.character.reparentTo(render)
        self.character.loadAnims({'walk': 'panda-walk'})
        self.character.loop('walk')
        self.character.hprInterval(15, Point3(360, 0,0)).loop()

It works fine with the shader Deyak posted, but does not work with the stock shader.

Ok, so the Nik dragon model I THINK is vertex coloured, and this issue might be being caused because the vertex colour isn’t been read correctly by the shader?

In Deyaks version the l_color doesn’t end up in the output, so the vertex colour isn’t passed along.

EDIT : Ignore this, see later post

Also, it seems theres some issues with vertex colours being passed along to shaders anyway. See: Passing vertex color to a shader

I got both the dragon and the standard textured ‘box’ appearing correctly by applying the above function by david:
python:

#Author: Kwasi Mensah
#Date: 7/11/2005
#
# This is a tutorial to show some of the more advanced things
# you can do with Cg. Specifically, with Non Photo Realistic
# effects like Toon Shading. It also shows how to implement
# multiple buffers in Panda.

import direct.directbase.DirectStart
from panda3d.core import PandaNode,LightNode,TextNode
from panda3d.core import Filename
from panda3d.core import NodePath
from panda3d.core import Shader
from panda3d.core import Point3,Vec4
from panda3d.core import BitMask32
from panda3d.core import InternalName
from panda3d.core import Geom
from panda3d.core import GeomVertexFormat
from panda3d.core import GeomVertexArrayFormat
from direct.task.Task import Task
from direct.actor.Actor import Actor
from direct.gui.OnscreenText import OnscreenText
from direct.showbase.DirectObject import DirectObject
from direct.showbase.BufferViewer import BufferViewer
import sys,os

# Function to put instructions on the screen.
def addInstructions(pos, msg):
    return OnscreenText(text=msg, style=1, fg=(1,1,1,1),
                        pos=(-1.3, pos), align=TextNode.ALeft, scale = .05)

# Function to put title on the screen.
def addTitle(text):
    return OnscreenText(text=text, style=1, fg=(1,1,1,1),
                        pos=(1.3,-0.95), align=TextNode.ARight, scale = .07)


class ToonMaker(DirectObject):
    def __init__(self):
        base.disableMouse()
        camera.setPos(0, -50, 0)
        
        # Check video card capabilities.

        if (base.win.getGsg().getSupportsBasicShaders() == 0):
            addTitle("Toon Shader: Video driver reports that shaders are not supported.")
            return

        # Post the instructions.
        self.title = addTitle("Panda3D: Tutorial - Toon Shading with Normals-Based Inking")
        self.inst1 = addInstructions(0.95,"ESC: Quit")
        self.inst2 = addInstructions(0.90,"Up/Down: Increase/Decrease Line Thickness")
        self.inst3 = addInstructions(0.85,"Left/Right: Decrease/Increase Line Darkness")
        self.inst4 = addInstructions(0.80,"V: View the render-to-texture results")

        # This shader's job is to render the model with discrete lighting
        # levels.  The lighting calculations built into the shader assume
        # a single nonattenuating point light.

        tempnode = NodePath(PandaNode("temp node"))
        tempnode.setShader(loader.loadShader("lightingGen.sha"))
        base.cam.node().setInitialState(tempnode.getState())
        
        # This is the object that represents the single "light", as far
        # the shader is concerned.  It's not a real Panda3D LightNode, but
        # the shader doesn't care about that.

        light = render.attachNewNode("light")
        light.setPos(30,-50,0)
                
        # this call puts the light's nodepath into the render state.
        # this enables the shader to access this light by name.

        render.setShaderInput("light", light)

        # The "normals buffer" will contain a picture of the model colorized
        # so that the color of the model is a representation of the model's
        # normal at that point.

        normalsBuffer=base.win.makeTextureBuffer("normalsBuffer", 0, 0)
        normalsBuffer.setClearColor(Vec4(1,1,1,0))
        self.normalsBuffer=normalsBuffer
        normalsCamera=base.makeCamera(normalsBuffer, lens=base.cam.node().getLens())
        normalsCamera.node().setScene(render)
        tempnode = NodePath(PandaNode("temp node"))
        tempnode.setShader(loader.loadShader("normalGen.sha"))
        normalsCamera.node().setInitialState(tempnode.getState())

        # Create a mask
        self.cartoon_mask = BitMask32.bit(1)

        # Apply our mask to the normals camera
        normalsCamera.node().setCameraMask(self.cartoon_mask)

        #what we actually do to put edges on screen is apply them as a texture to 
        #a transparent screen-fitted card

        drawnScene=normalsBuffer.getTextureCard()
        drawnScene.setTransparency(1)
        drawnScene.setColor(0,0,0,0)
        drawnScene.reparentTo(render2d)
        self.drawnScene = drawnScene

        # this shader accepts, as input, the picture from the normals buffer.
        # it compares each adjacent pixel, looking for discontinuities.
        # wherever a discontinuity exists, it emits black ink.
                
        self.separation = 0.001
        self.cutoff = 0.3
        inkGen = loader.loadShader("inkGen.sha")
        drawnScene.setShader(inkGen)
        drawnScene.setShaderInput("separation", Vec4(self.separation,0,self.separation,0));
        drawnScene.setShaderInput("cutoff", Vec4(self.cutoff,self.cutoff,self.cutoff,self.cutoff));
        
        # Panda contains a built-in viewer that lets you view the results of
        # your render-to-texture operations.  This code configures the viewer.

        self.accept("v", base.bufferViewer.toggleEnable)
        self.accept("V", base.bufferViewer.toggleEnable)
        base.bufferViewer.setPosition("llcorner")


        self.character = Actor()
        self.character.loadModel('models/nik-dragon.egg.pz')
        self.makeFloatColors(self.character)
        self.character.flattenStrong()
        self.character.reparentTo(render)

        self.character.hprInterval(15, Point3(360, 0,0)).loop()

        self.character2 = loader.loadModel('box')
        self.character2.reparentTo(render)
        self.character2.setX(10)

        # Hide character2 from the normal camera
        #self.character2.hide(self.cartoon_mask)


        self.character.hprInterval(15, Point3(360, 0,0)).loop()
        self.character2.hprInterval(15, Point3(360, 0,0)).loop()

        # these allow you to change cartooning parameters in realtime

        self.accept("escape", sys.exit, [0])
        self.accept("arrow_up", self.increaseSeparation)
        self.accept("arrow_down", self.decreaseSeparation)
        self.accept("arrow_right", self.increaseSeparation)
        self.accept("arrow_left", self.decreaseSeparation)

    def increaseSeparation(self):
        self.separation = self.separation * 1.11111111;
        print self.separation
        self.drawnScene.setShaderInput("separation", Vec4(self.separation,0,self.separation,0));
        
    def decreaseSeparation(self):
        self.separation = self.separation * 0.90000000;
        print self.separation
        self.drawnScene.setShaderInput("separation", Vec4(self.separation,0,self.separation,0));
        
    def increaseCutoff(self):
        self.cutoff = self.cutoff * 1.11111111;
        print self.cutoff
        self.drawnScene.setShaderInput("cutoff", Vec4(self.cutoff,self.cutoff,self.cutoff,self.cutoff));
        
    def decreaseCutoff(self):
        self.cutoff = self.cutoff * 0.90000000;
        print self.cutoff
        self.drawnScene.setShaderInput("cutoff", Vec4(self.cutoff,self.cutoff,self.cutoff,self.cutoff));
        

    def makeFloatColors(self, nodePath):
        colorName = InternalName.getColor()

        vdMap = {}

        # Get all the GeomVertexDatas at this NodePath and below.
        for gnp in nodePath.findAllMatches('**/+GeomNode').asList():
            gn = gnp.node()
            for i in range(gn.getNumGeoms()):
                geom = gn.modifyGeom(i)
                vd = geom.getVertexData()
                newVd = vdMap.get(vd, None)

                if newVd == None:
                    # Check the format of this GeomVertexData.
                    format = vd.getFormat()
                    color = format.getColumn(colorName)
                    if color and color.getNumericType == Geom.NTFloat32:
                        # This data already has a color column, and it's
                        # already a float type.  Great!
                        pass

                    else:
                        # We need to convert an existing column column to
                        # float, or add a new float color column.

                        newFormat = GeomVertexFormat(format)
                        newFormat.removeColumn(colorName)

                        colorArray = GeomVertexArrayFormat(colorName, 4, Geom.NTFloat32, Geom.CColor)
                        newFormat.addArray(colorArray)
                        newFormat = GeomVertexFormat.registerFormat(newFormat)

                        newVd = vd.convertTo(newFormat)
                        vdMap[vd] = newVd

                if newVd != None:
                    geom.setVertexData(newVd) 

t=ToonMaker()

run()

lightingGen.sha:

//Cg
//
//Cg profile arbvp1 arbfp1

void vshader(in float2 vtx_texcoord0 : TEXCOORD0,
             out float2 l_texcoord0 : TEXCOORD0,
             float4 vtx_position   : POSITION,
             float3 vtx_normal     : NORMAL,
             float4 vtx_color      : COLOR,
             out float4 l_position : POSITION,
             out float4 l_brite,
             out float4 l_color    : COLOR,
             uniform float4 mspos_light,
             uniform float4x4 mat_modelproj)
{
  l_position = mul(mat_modelproj, vtx_position);
  float3 N = normalize(vtx_normal);
  float3 lightVector = normalize(mspos_light - vtx_position);
  l_brite = max(dot(N,lightVector), 0);
  l_color = vtx_color;
  l_texcoord0 = vtx_texcoord0;
}


void fshader(in float2 l_texcoord0 : TEXCOORD0,
             uniform sampler2D tex_0 : TEXUNIT0,
             float4 l_brite,
             in float4 l_color     : COLOR0,
             out float4 o_color : COLOR0)
{
  if (l_brite.x<0.5) l_brite=0.8;
  else l_brite=1.2;
  o_color = l_brite * l_color;
  float4 texcolor = tex2D( tex_0, float2( l_texcoord0) );
  if(texcolor.x){
	o_color *= texcolor;
  }
}

Note that the shader if statement sucks, but the Dragon doesn’t seem to pass a texture unlike the normal box, hence the if statement - alternatives welcome :smiley:

Does it work if you multiply the value of vtx_color with (1.0 / 255.0) ?

:blush:

Yes - yes it does.

So the python code I posted isn’t really needed. A flattenStrong() on the dragon gets the eyes working, and adjusted the shader with a scaled vtx_color.

//Cg
//
//Cg profile arbvp1 arbfp1

void vshader(in float2 vtx_texcoord0 : TEXCOORD0,
             out float2 l_texcoord0 : TEXCOORD0,
             float4 vtx_position   : POSITION,
             float3 vtx_normal     : NORMAL,
             float4 vtx_color      : COLOR,
             out float4 l_position : POSITION,
             out float4 l_brite,
             out float4 l_color    : COLOR,
             uniform float4 mspos_light,
             uniform float4x4 mat_modelproj)
{
  l_position = mul(mat_modelproj, vtx_position);
  float3 N = normalize(vtx_normal);
  float3 lightVector = normalize(mspos_light - vtx_position);
  l_brite = max(dot(N,lightVector), 0);
  l_color = vtx_color * (1.0 / 255.0);
  l_texcoord0 = vtx_texcoord0;
}


void fshader(in float2 l_texcoord0 : TEXCOORD0,
             uniform sampler2D tex_0 : TEXUNIT0,
             float4 l_brite,
             in float4 l_color     : COLOR0,
             out float4 o_color : COLOR0)
{
  if (l_brite.x<0.5) l_brite=0.8;
  else l_brite=1.2;
  
  float4 texcolor = tex2D( tex_0, float2( l_texcoord0) );
  
  // Apply vertex colours 
  o_color = l_color;

  // If we have a texture, lets apply that instead
  if(texcolor.x){
	o_color = texcolor;
  }
  
  // Apply final lighting
  o_color *= l_brite;
  
}

Right, so the problem is that (1) vtx_color is not rescaled from 0-255 to 0-1 range and (2) .

Could you please try and see what happens when you set “basic-shaders-only #t” in Config.prc?

These two combinations give correct output:

“basic-shaders-only 1” + l_color = vtx_color;
“basic-shaders-only 0” + l_color = vtx_color * (1.0 / 255.0);

So, the vtx_color only needs to be scaled when basic-shaders are off.