advanced cartoon shader tutorial problem[solved]

Hi friends, I used the advanced cartoon shader tutorial code and shader files in my game, but the results arent what I expected:

My model is exported from blender, and during runtime some vertex colored parts and all the textured part of the model are shown white. When I switch the manual lighting shader for the automatic, everything is allright. So probably the problem is the tutorial lighting shader file, what could be wrong with this code?

Thanks in advance

The bottom line is, nothing’s wrong with the code per se; but the shader as written simply ignores your texture. If you want your texture to be included, you need to add code to the shader to respect the texture.


well, then the problem was which the code was lacking texture support isn it drwr?

I found this code by josh yelon, but it seems it is reading only texture color, not vertex color. That raises another question I have:

A model vertex colored is more economic for the computer to process than being textured, or being textured or vertex colored do not matter, since the difference is small?

thanks for the help drwr

i’d say, if your gpu is new enough to handle shaders the difference between vertex and texture colors is neglectable (textures might use a bit more texture memory but that doesnt really give a speed penalty)

Hey! another Ubuntu user hehe

Its good to know the diference is minimal, and about the code above, any ideia how to make it accept both vertex colored and textured parts of the model? The thing is, my model is vertex colored at face, hair, hands, legs and feet and texture colored in everything else( which needed more detail without increasing polys ).

Thanks in advance, I will continue to experiment here of course

Sounds like you need a good shader tutorial.


Ah, I have already been reading both lately: the nvidia Cg tutorial and Azraiyl tutorials.

I have understood some concepts already, but still I lack some important knowledge. But its ok, I will ready more before making any further questions.

Thanks for the help, will post the cg code here if I suceed

well, Im managing to understand cg better as weeks pass, but, well… I think I still seriously need help here…

I cant conciliate vertex color and texture colors at same time the shader. I only managed to get one at time to work, not both…by both I mean to somehow detect the textured parts of the model which would receive texcolor * l_texcoord0( as shown below ) and the vertex colored parts to receive l_color * l_texcoord0…i dunno

the above is the vertex color working one( and yet working very ugly ), the lines marked as comment if uncommented( and comment other parts of the shader ) let the shader accept the texture but not vertex colors…but I dont get it very well…

Im curious about how the shader generator shaders manage to get vertex colors and texture colors along simultanealy and so perfectly…

Thanks in advance for any light at the matter

To apply vertex colors and texture colors at the same time, you need to multiply the texture color and the vertex color and assign that as the output color.


Also, you’re using the result of vertex lighting as texture coordinates?

If you want to see the shaders that the Autoshader is producing, you can add ‘dump-generated-shaders #t’ to your config file. There’s a way to pull the pre-compiled shader text from the auto-shader generator itself, but I don’t recall where the hook to this lies. But IIRC, it amounts of grabbing the shader generator object, creating a new shaderAttrib and having it generate the shader, which you can then easily extract. Check the manual/forums for the exact method, because I don’t remember it. :slight_smile:

This might help if you want to examine what the Autoshader is doing and then figure out how to modify it for your own needs.

Aside from that, I would find a tutorial or book that covers the basics of 3d realtime lighting. This is pretty well documented online, such as in the OpenGL Red Book. The math behind the lighting calculations for OpenGL at there for the fixed function pipeline. Once you understand this, then seeing what shaders are doing should make more sense.

Have finished reading Azraiyl tutorial today and will be reading more of the nvidia cg tutorial and the Opengl red book soon.

The cg code until now is below:

if I use o_color = texcolor * toonlight( just to see what it would like), i get

if I use o_color = l_color * toonlight

if I use o_color = l_color * texcolor * toonlight

since Im a begginer in cg, and everything worked allright with shader generator, then these problems are probably located in my shader and are my fault.

Not sure why, but shader generator gave me 7 shaders when I was only using lightramp and a ambiental light and a point light.

have been making a mistake with tex_color = tex2D. Its allright now. I got lighting working for both texcolor and vertexcolor simultaneous. Only need to discover why my vertex colors are messed up when combined with texcolors( which are almost the way I want )…

havent had time for reading more about cg, things are a bit busy here

I have seen the dumped shader generator shaders( 5 of them ) code for this model without any aditional lights( just the default one ), and even used each one to see the results, but it didnt helped me much…it seems the code each one have is already include in my current cg code. yet, the shader generator works fine, and my code not…

Lighting is allright, texture is allright, the only problem is the vertex color, which is all weird( look the post above ) alone or multiplied with texcolor( even worse ).

Anyone has an ideia about what is happening?

Try applying the code from the generated shader exactly. Presumably that will work. If it does, then try making small changes, one at a time, until it is identical to the shader you have written. Eventually you should see the change that breaks it.


If you are using an Nvidia card, you can also use the Nvidia composer tools to do shader debugging. I’m not at all familiar with this, but there should be a tool for ATI cards along those lines too.

FX Composer generates CgFX shaders, and as they are ATM not supported by Panda, you’d need to manually extract the Cg stuff from them, which is probably trickier than writing a shader yourself.

Well, I gave up in the vertex colors and texture colors simultaneous for now. I´m using 1x1 pixel textures with a single color each to replace vertex colors. If you´re into texture colors only, here´s the cg code:

I have also noticed something very important for people using blender to make their 3d. If your lighting looks weird, not smooth at all, you need only to select the problematic faces press w(on edit mode of course) and select set smooth, and your problem is likely to dissapear.

Ehrm, if you want a solid color for the whole model, you don’t need to use vertex colors, you can use the color attribute of a node (available in a shader via attr_color, settable using nodePaht.setColor)

I think you misunderstood me rdb. I wanted to use only texture colors in the textured parts of the model and to use vertex color( solid colors ) on the untextured and vertex colored parts of the model, both of them at the same time, since the model has both, texture colors and vertex colors.

Vertex color for things like hair mesh, headgear and such, texture colors for things which need more than a solid color( cloth, boots ). I must say I´m going for a cartoonish look to my game.

But when together the result is quite ugly. I have dumped the shader generator shaders files( it gave me 6 of them at once, for one object with cartoon style lighting ), looked the code and still I haven´t suceeded. I have even used one of the shader files intouched in my game and it didn´t worked. The funny thing is when I let the shader generator do the job( automatically ), texture colors and vertex colors are recognized simultaneous.

Its funny how each of the shaders files I saw for a single object, colored a diferrent part of this object( one was vertex colored, other was texture colored ), I have no ideia why.

But I found a workaround. I used only texture colors and in the parts of the model which where of only one color( where I used diferrent vertex colors before), like the purple hair and blue headgear, I used 1x1 pixel texture with just one color. This texture size is probably nothing to the video card.

It´s allright now, the model is perfectly shaded. The only thing is I´m using only textures now, not vertex colors.

And thanks for your help pro-soft, I´m gratefull for it as I am for the help of drwr, Thomas Egi and FenrirWolf.

Hi, I have thought of something which might be usefull. I´m now using a “if” statement, I´m a bit confused about how would I go to detect if a texture exists for a fragment being rendered, but until now I managed to get vertex colors and texture colors partially working at the same time.

float4 finalcolor;
float4 texcolor = tex2D( tex_0, float2( l_texcoord0 ) );
float3 lightpos =;
float3 modelpos = l_myposition;
float3 normal = normalize( l_mynormal );
float3 direction = normalize( lightpos - modelpos );
float toonlight = max( dot( normal, direction ), 0 ) ;
if ( toonlight < 0.5 ) toonlight = 0.8;
else toonlight = 1;
if ( texcolor.x > 0 ) o_color = texcolor * toonlight;
else o_color = l_color * toonlight;

The ideia was which if a texture exists, then texture colors would output to screen, else, if no texture exists, then vertex colors would be used. I should use something other than texcolor.x since transparent parts of my texture are output as white, and there are some other issues happening.

I think shader generator had one shader for texture and other shader for vertex colors because it could apply each shader where they were needed, so if a part had texture, it would use the texture shader, if not, it would use the vertex color shader. Anyway I´m not going to do this, I will try to get both into a single shader file, if this is not possible i will soon know.

Teedee told me which it is better to use diferrent shaders for diferent colored models, because it is better.