Need Help Porting a old RimLight Shader

Hello everyone, I wanted to ask some questions on panda3d GLSL and CG shader code differences below is a old Cg shader that I use a lot, I read that panda3d will be soon dropping support for the Cg shading language.

Not wanting to lose the look of my game I have decided to a take GLSL language course so I can port the shader myself

//Cg

void vshader(
             float4 vtx_position : POSITION,
             float2 vtx_texcoord0 : TEXCOORD0,
             float3 vtx_normal : NORMAL,
             uniform float4x4 mat_modelproj,
             uniform float4x4 trans_model_to_world,
             uniform float4 mspos_cam,
             uniform float4 AmbLight,

             out float  l_smooth,
             out float  l_facingRatio,
             out float2 l_texcoord0 : TEXCOORD0,
             out float3 l_texcoord1 : TEXCOORD1,
             out float4 l_position : POSITION
             )
{
  l_smooth = smoothstep( AmbLight,1,dot(vtx_normal, normalize(mspos_cam-vtx_position)) );
  l_facingRatio = pow( 1.0-saturate( dot(vtx_normal, normalize(mspos_cam-vtx_position)) ), 4 );
  l_position = mul(mat_modelproj, vtx_position);
  l_texcoord0=vtx_texcoord0;

  l_texcoord1=mul(trans_model_to_world, float4(vtx_normal,0)).xyz;
}

void fshader(
             in float2 l_texcoord0: TEXCOORD0,
             in float3 l_texcoord1: TEXCOORD1,
             in float  l_smooth,
             in float  l_facingRatio,
             uniform float4 k_DirLight,
             uniform samplerCUBE k_envirCubeMap,
             sampler2D tex_0 : TEXUNIT0,

             out float4 o_color:COLOR)
{
  float4 tex = tex2D(tex_0, l_texcoord0);
  float3 cubeMap = texCUBE(k_envirCubeMap,l_texcoord1).xyz * k_DirLight.x;
  o_color = float4( lerp(tex.rgb*l_smooth, (tex.rgb+cubeMap)*.5, l_facingRatio) , tex.a );
}

I actually got the shader from a vary old thread here Mario Galaxy-style velvet shader? - General Discussion - Panda3D now, my attempt is below, the vert shader.

#version 150 // vertex shader

uniform mat4 p3d_ModelViewProjectionMatrix;
uniform mat4 p3d_ModelMatrix;
uniform vec4 mspos_cam;
uniform vec4 AmbLight;

in vec4 p3d_Vertex;
in vec2 p3d_MultiTexCoord0;
in vec3 p3d_Normal;

out float l_smooth;
out float l_facingRatio;
out vec2 li_TexCoord0;
out vec3 li_TexCoord1;

void main() {
    vec3 lhtsrc = mspos_cam.xyz;
    l_smooth = smoothstep(AmbLight[0], 1.0, dot(p3d_Normal, normalize(lhtsrc - p3d_Vertex.xyz)));
    vec3 viwsrc = mspos_cam.xyz;
    l_facingRatio = pow(1.0 - clamp(dot(p3d_Normal, normalize(viwsrc - p3d_Vertex.xyz)), 0.0, 1.0), 4.0);
    vec4 l_position = p3d_ModelViewProjectionMatrix * p3d_Vertex;
    gl_Position = l_position;
    li_TexCoord0 = p3d_MultiTexCoord0;
    li_TexCoord1 = vec3(p3d_ModelMatrix * vec4(p3d_Normal, 0.0));
    
}

then the frag shader

#version 150 // fragment shader

uniform vec4 DirLight;
uniform samplerCube envirCubeMap;
uniform sampler2D p3d_Texture0;

in float l_smooth;
in float l_facingRatio;
in vec2 li_TexCoord0;
in vec3 li_TexCoord1;

out vec4 FragColor;

vec4 tex;
vec3 mixed;

void main() {
    vec4 tex = texture(p3d_Texture0, li_TexCoord0);
    vec3 cubeMap = texture(envirCubeMap, li_TexCoord1).rbg * DirLight[0];
    vec3 mixed = (tex.rgb * l_smooth) * (1.0 - l_facingRatio) + ((tex.rgb + cubeMap) * .5) * l_facingRatio;
    FragColor = vec4(mixed, tex.r);
}

the questions I wanted to ask was

1: is mat_modelproj and trans_model_to_world something in the control program? I ask because there are no shader inputs for them and no Cg commands for them in the list, yet they somehow work (this has been solved by Thaumaturge)

2: I need to know how to get nodes into the shader and also get their position like the original Cg shader, it was done with a shader input for the node then in the shader itself, uniform float4 (vec4 for GLSL) mspos_ then the shader input name (solved by Thaumaturge, but waiting confirmation)

3: a little embarrassing, but is there GLSL command equivalent to Cg mul()? (doing some research, I found this to be multiplying matrices by a vec3, which was completely over my head and even glsl has trouble doing this)

a thank you to anyone who could help, have a good one guys

The “trans_model_to_world” matrix is mentioned in the manual, I believe, but not explicitly. Instead, it lists “uniform float4x4 trans_x_to_y”, where “x” and “y” are coordinate spaces–in this case, being “model”(-space) and “world”(-space).

The “mat_modelproj” matrix is, I think, mentioned explicitly on the same page.

As far as I’m aware, the same should work for GLSL, just with GLSL syntax, like so:

uniform vec3 mspos_<the given name of the node>

ah, you are correct I overlooked the mat_modelproj, and the trans_model_to_world went over my head (laughs), thanks I have reflected the corrections in the 1st post, I also relfected your node solution, but because I did not see in the list, I want to wait in for someone to confirm it is correct, thank you for your suggestions, they have helped.

1 Like

so I tired the shaders and my character goes darken, not being able to debug the shaders, I tried to purposely make a mistake in the shader, and yeah, same darkness in the character, so I assume the shaders crashed.

which is fine being its my first attempt, however as I said above, I cannot debug the shader, I know there are notify options, but that information does not appear in my shell/console, the thing is I use IDLE, so you have to use print() for it to show up there.

I was wondering if there was a way to get Notify to print to console, or to get the openGL debug information or error messages to print to console, thanks for listening

EDIT: I found out how to display the error messages, i can be done with this code here just before you declare showbase

from panda3d.core import loadPrcFileData, MultiplexStream, Notify
loadPrcFileData("", "notify-level-glgsg error")
nout = MultiplexStream()
nout.addFile("glerror.txt")
Notify.ptr().setOstreamPtr(nout, 0)

so after debugging for a day I finally got it to a point where no error messages where displaying, however, my character is now invisible, I have decided that this port is way beyond my head, so I will end this thread and give the solution to thuamaturge. thanks

Ah, I’m sorry to read that it hasn’t worked out! :/

Something that I find useful in debugging shaders, if I may, is to have the shader temporarily output intermediate values as its colours. That way one can potentially examine what’s happening along the course of the shader, even though the printing of values to the console may not be available.

For example, let’s say that we had the following GLSL fragment shader:

in vec2 texCoord;

uniform sampler2D p3d_Texture0;

uniform vec3 mspos_someNodePath;

out vec4 color;

void main()
{
    vec4 pixel = texture(p3d_Texture0, texCoord);
    
    float distance = length(mspos_someNodePath);

    color = pixel * sin(distance);
}

Now, let’s say that the above wasn’t doing what we wanted it to do. We might decide that we want to see what sort of values I’m getting in the “distance” variable, and so make the following temporary change:

in vec2 texCoord;

uniform sampler2D p3d_Texture0;

uniform vec3 mspos_someNodePath;

out vec4 color;

void main()
{
    vec4 pixel = texture(p3d_Texture0, texCoord);
    
    float distance = length(mspos_someNodePath);

    //color = pixel * sin(distance);
    color.xyz = vec3(0, distance, 0);
    color.w = 1;
}

This causes the shader to render its surface as fully opaque (making the results visible), and more green the higher the value of “distance”. (And if that value doesn’t fit comfortably between zero and one–if for example it increases too quickly–we might multiply it by a some appropriate number to bring it into that zero-to-one range.)

it was a little much for me to take a program I knew next to nothing about and port it to something I barely can grasp at (Since I learnt glsl a couple of days aga) as my vary first shader.

the thing is my instructor made a couple of shader examples during the course, one which was a rimlight and a cube example (much similar to this shader, except reversed), that I intend on using as my plan B :grin:

the down side to glsl is I had to force GL version 3.2 using a PrcFile call and that killed the autoshader, the good thing is, I used auto shader mostly only for normal mapping, and there is a normal map shader example in the course I can use to replace that.

however, the blend transparency (glass and water surfaces) is something I have to figure out myself or seek out some kind of help, anyway, I will keep your suggestion in mind for degugging, thanks

Fair enough, on all counts! And indeed, the suggestion is somewhat of a general one, something that, even if it doesn’t help here, may well help in the future!

Well then, good luck with your future shader-programming! :slight_smile:

1 Like

thanks, you know, your suggestion probably would have assisted in this shader, its just, I was really counting on this shader being straight forward and just working once I got the equivalent variables and commands right.

There is trigonometry and possibly calculous calculations in this shader, I found out as I got into more research, something way beyond my caveman level of programming (laughs)

1 Like

I mean, the actual logic and calculation should be just as in the original, I would imagine. It seems to me fairly likely that there’s just some oversight in the porting of it. And, well, this is where such debugging might help!

(For example, you could make equivalent changes to both shaders and see whether they respond in the same way. If they don’t, then there might be a discrepancy there that could cause a problem in the final result of the shader!)

Naturally, if you’re struggling with following the shader, then this might be less effective. Conversely, debugging it in this way may actually help with understanding the shader!

hmm, I guess I can give quick whack at it tomorrow night, but the logic is not completely intact though, only a assumed equivalent, that could be the problem, like the mul() command for Cg I have found no straight equivalent, just vague explanations on what it does and nebulous answers on how to pull it off in glsl

1 Like

On that, take a look at the page linked-to below–it seems to indicate that GLSL does, in fact, have “mul”, but that its order or parameters is reversed:

1 Like

oh wow, I did not know this exist, thank you, I will take a look at it.

edit: alright, I will study the page and continue this tomorrow night then and post my findings here then, thanks for the help

1 Like

Not a problem! Good luck, and have fun! :slight_smile:

1 Like

alright, I am back and tried your suggestion, and the character is still invisible, this might Indicate that the issue has nothing to do with the shader but maybe something to do with my graphics since your suggestion just simply changes the color.

I am going to do more research to see if I can fix this.

1 Like

I might suggest trying the simplest possible GLSL shader–just one that does nothing but assign a colour–and seeing how that works.

I might also suggest checking your vertex-shader–perhaps the model is less “invisible” than “incorrectly transformed, and thus out of view”.

[edit]
Actually, yes, looking over your vertex shader above once more I notice now that you don’t seem to be writing to “gl_Position”, and thus aren’t transforming your vertices. (I’m not sure at all of where they end up if you don’t write to that variable.) You also seem to be writing to something called “p3d_Position”, which I’m not finding information on…

Hmm… Perhaps it would help if I gave you a very basic sample-shader to illustrate the fundamentals:

Vertex shader:

#version 130
// The above is just the version that I usually work with, and
// have to hand the syntax for. Alterations may be called for
// with other versions.

in vec4 p3d_Vertex; //The current vertex of the model

uniform mat3 p3d_ModelViewProjectionMatrix;
// The matrix that transforms from model-space to final render-space

void main()
{
    gl_Position = p3d_ModelViewProjectionMatrix * p3d_Vertex;
    // By writing to "gl_Position", we place the model's vertices
    // in position to be rendered
}

Fragment shader:

#version 130
// As above, this is just the version that I'm familiar with

out vec2 color;
// This is, essentially, the final output of the shader;
// a special variable that holds the rendered fragment-value

void main(){
    vec3 colour = vec3(0, 1, 0); // I like green. :P
    
    color.xyz = colour; // Set the model's colour
    color.w = 1; // Make the model opaque
}

(This is, please note, untested; I may have missed some bugs!)

Actually I solved the problem, thanks to your first suggestion, it was my vertex position, I was feeding it p3d_position believing it was going into panda3d’s graphics, when I should have been feeding it to gl_position, I was relived when I saw the green character, then I plugged in the original code and there it was the, rim shader
WIN_20230220_01_23_08_Pro
You can see the effect around the outline of head, thank you man, the shader works!

1 Like

Ah, excellent! I’m glad that you got it working in the end! :slight_smile:

1 Like

Yes and time to expand upon it, :grin: i will reflect the changes in the 1st post, again thanks for the help :+1:

1 Like