Overlapping texture masks [shader]

I’m working on a terrain texturing shader. I’ve got a total of 6 textures - 5 ‘real’ textures and one that has a mask for each of these textures in on channel. The last texture has no mask of its own, just a mix of the other 4 masks.

     result.rgba *= tex0.rgba*(1-texA.r)*(1-texA.g)*(1-texA.b)*(1-texA.a);
     result += texCob*texA.r;
     result += texSand*texA.g;
     result += texRock*texA.b;
     result += texStone*texA.a;

‘tex0’ is the base texture(grass) with no mask of it’s own, ‘texA’ is the mask and ‘texCob’, ‘texSand’, ‘texRock’ and ‘texStone’ are the real textures.

It works almost good, but I’ve got one problem - in some places the masks overlap each other. I’ve got a spot where my cobble mask is not zero and my sand mask in not zero and in that spot both textures get multiplied giving a over-saturated, bright artifact.
You can see it on this image:

On the top one the masks are color-coded (red =sand, green=cobble), the bright spot is where they overlap (kind of yellow-ish).

If I multiply the layers with a negative of the masks not used for that layer e.g.:

result += texCob*texA.r*(1-texA.g)*(1-texA.b)*(1-texA.a);

I end up with the problem spots with no texture at all (and a few less fps :unamused: ).

Are there any known tricks for situations like this?

I haven’t checked this and so may well be mistaken, but could you perhaps multiply all of the mask channel values together and then in the accumulation divide each by that value, essentially normalising the mask?

Something like this:

     # I'm presuming that multiplications are at least a little faster than divisions,
     #  so I'm arranging to divide only once.
     scalar = 1.0/texA.r*texA.g*texA.b*texA.a
     result.rgba *= tex0.rgba*(1.0-scalar);
     result += texCob*texA.r*scalar;
     result += texSand*texA.g*scalar;
     result += texRock*texA.b*scalar;
     result += texStone*texA.a*scalar;

Either I’m doing something wrong (the scalar should be of type ‘float’?) or it won’t work like that. I’m getting holes where textures should be.

There should be cobbles on the top one (there’s just the normal map) and a mix of rock,stone and sand where the white gaps with colored dots are.

Ah, wait, checking myself that does look incorrect. For one thing you should end up with a division by zero when there is nothing but grass… :confused:

Looks like the part of the shader that I posted was wrong anyway, the extra textures did not have lighting.
I came up with this:

result.rgba *=tex0*(1-texA.r)*(1-texA.g)*(1-texA.b)*(1-texA.a)

It works, but just because I know that cobbles ans sand overlap so I’m multiplying the cobbles with the negative from the sand mask and doing the same with the others.
If I make a new map where cobbles are on stone I will get the artifacts again.

I’m starting to think that trying to fix this in the shader is a bad idea, maybe I should sum, divide or multiply the channels of the mask-texture before I even send it to the shader. Can’t wrap my brain around what I should actually do :neutral_face:

Oh, and here’s the shader I’m using, all of it. I have little to no idea of what I’m actually doing, so if something looks wrong it probably is wrong:

/* Generated shader for render state 09E41304:
  TexMatrixAttrib: shadow(T:(identity)) ts(T:(identity))
  TextureAttrib:on default:gras5 ts:gras5_n shadow: ts:fog3
  TexGenAttrib: shadow(world_position) ts(world_position)
void vshader(
	 in float4 vtx_texcoord0 : TEXCOORD0,
	 out float4 l_texcoord0 : TEXCOORD0,
	 uniform float4x4 trans_model_to_world,
	 out float4 l_world_position : TEXCOORD4,
	 uniform float4x4 trans_model_to_view,
	 out float4 l_eye_position : TEXCOORD5,
	 uniform float4x4 tpose_view_to_model,
	 out float4 l_eye_normal : TEXCOORD6,
	 in float4 vtx_normal : TEXCOORD4,
	 out float4 l_tangent : TEXCOORD1,
	 out float4 l_binormal : TEXCOORD2,
	 float4 vtx_position : POSITION,
	 out float4 l_position : POSITION,
	 uniform float4x4 mat_modelproj
) {
     float3 vtx_tangent1= float3(vtx_normal.x, vtx_normal.z, -vtx_normal.y);
	 float3 vtx_binormal1= float3(vtx_normal.z, vtx_normal.y, -vtx_normal.x); 
	 l_position = mul(mat_modelproj, vtx_position);
	 l_world_position = mul(trans_model_to_world, vtx_position);
	 l_eye_position = mul(trans_model_to_view, vtx_position);
	 l_eye_normal.xyz = mul((float3x3)tpose_view_to_model, vtx_normal.xyz);
	 l_eye_normal.w = 0;
	 l_texcoord0 = vtx_texcoord0;
	 l_tangent.xyz = mul((float3x3)tpose_view_to_model, vtx_tangent1.xyz);
	 l_tangent.w = 0;
	 l_binormal.xyz = mul((float3x3)tpose_view_to_model, -vtx_binormal1.xyz);
	 l_binormal.w = 0;

void fshader(
	 in float4 l_world_position : TEXCOORD4,
	 in float4 l_eye_position : TEXCOORD5,
	 in float4 l_eye_normal : TEXCOORD6,
	 uniform sampler2D tex_0,
	 in float4 l_texcoord0 : TEXCOORD0,
	 uniform sampler2D tex_1,
	 uniform sampler2D tex_2,
	 uniform float4x4 texmat_2,
	 uniform sampler2D tex_3,
        in uniform sampler2D mask_texture,
        in uniform sampler2D rock,
        in uniform sampler2D rock_n,
        in uniform sampler2D cobble,
        in uniform sampler2D cobble_n,
        in uniform sampler2D sand,
        in uniform sampler2D sand_n,
        in uniform sampler2D stone,
        in uniform sampler2D stone_n,
	 uniform float4x4 texmat_3,
	 in float3 l_tangent : TEXCOORD1,
	 in float3 l_binormal : TEXCOORD2,
	 uniform float4 alight_alight0,
	 uniform float4x4 dlight_dlight0_rel_view,
	 out float4 o_color : COLOR0,
	 uniform float4 attr_color,
	 uniform float4 attr_colorscale
) {
	 float4 result;
	 float4 l_texcoord2 = l_world_position;
	 l_texcoord2 = mul(texmat_2, l_texcoord2);
	 l_texcoord2.xyz /= l_texcoord2.w;
	 float4 l_texcoord3 = l_world_position;
	 l_texcoord3 = mul(texmat_3, l_texcoord3);
	 l_texcoord3.xyz /= l_texcoord3.w;
	 // Fetch all textures.
	 float4 tex0 = tex2D(tex_0, l_texcoord0.xy*64);
	 float4 tex1 = tex2D(tex_1, l_texcoord0.xy*64); //normal map
	 float4 tex2 = tex2D(tex_2, l_texcoord2.xy);
	 float4 tex3 = tex2D(tex_3, l_texcoord3.xy);
     float4 texA = tex2D(mask_texture, l_texcoord0.xy);
     float4 texRock=tex2D(rock, l_texcoord0.xy*64);
     float4 texRock_N=tex2D(rock_n, l_texcoord0.xy*64);
     float4 texCob=tex2D(cobble, l_texcoord0.xy*64);
     float4 texCob_N=tex2D(cobble_n, l_texcoord0.xy*64);     
     float4 texSand=tex2D(sand, l_texcoord0.xy*64);
     float4 texSand_N=tex2D(sand_n, l_texcoord0.xy*64);  
     float4 texStone=tex2D(stone, l_texcoord0.xy*64);
     float4 texStone_N=tex2D(stone_n, l_texcoord0.xy*64);  
     float4 tex_normal=tex1*(1-texA.r)*(1-texA.g)*(1-texA.b)*(1-texA.a)
	 // Translate tangent-space normal in map to view-space.
	 float3 tsnormal = ((float3)tex_normal * 2) - 1;
	 l_eye_normal.xyz *= tsnormal.z;
	 l_eye_normal.xyz += l_tangent * tsnormal.x;
	 l_eye_normal.xyz += l_binormal * tsnormal.y;
	 // Correct the surface normal for interpolation effects
	 l_eye_normal.xyz = normalize(l_eye_normal.xyz);
	 // Begin view-space light calculations
	 float ldist,lattenv,langle;
	 float4 lcolor,lspec,lvec,lpoint,latten,ldir,leye,lhalf;
	 float4 tot_ambient = float4(0,0,0,0);
	 float4 tot_diffuse = float4(0,0,0,0);
	 // Ambient Light 0
	 lcolor = alight_alight0;
	 tot_ambient += lcolor;
	 // Directional Light 0
	 lcolor = dlight_dlight0_rel_view[0];
	 lspec  = dlight_dlight0_rel_view[1];
	 lvec   = dlight_dlight0_rel_view[2];
	 lcolor *= saturate(dot(l_eye_normal.xyz, lvec.xyz));
	 tot_diffuse += lcolor;
	 // Begin view-space light summation
	 result = float4(0,0,0,0);
	 result += tot_ambient;
	 result += tot_diffuse;
	 result = saturate(result);
	 // End view-space light calculations
	 result.a = 1;
     result.rgba *=tex0*(1-texA.r)*(1-texA.g)*(1-texA.b)*(1-texA.a)
	 result.rgb = lerp(result, tex2 * float4(0, 0, 0, 1), tex2.r).rgb;
	 result.rgba *= tex3.rgba;
	 result *= attr_colorscale;     
	 o_color = result * 1.000001;

I’m glad that you’ve made progress! :slight_smile:

Here’s what I’ve come up with in the meanwhile, based on your originally-posted code, and thus without consideration to lighting:

All right, let’s try this again – and let me check myself this time. ^^;;;

I gather that you want the following:
[] Grass shows through wherever nothing else is set to be showing.
] Where something else is showing, all things showing should blend together.
[*] If something else is showing, and its mask is 1, no grass should be visible.

I’m not terribly familiar with shaders – do you have access to “if” statements? I’m presuming so for the moment. I’m also presuming that “result” starts off zeroed.

     total = texA.r+texA.g+texA.b+texA.a
     if total > 0:
          scalar = 1.0
          if total > 1.0:
               scalar = 1.0/total
          result += texCob*texA.r*scalar;
          result += texSand*texA.g*scalar;
          result += texRock*texA.b*scalar;
          result += texStone*texA.a*scalar;
     if total < 1.0:
          result += tex0*(1.0-total);

So, when there is nothing but grass:

total = 0+0+0+0 = 0
total is not greater than 0, so skip the first “if”
total is less than 1, so run the second “if”:
result += tex0*(1-0) = tex0

So we should see only grass.

When there are cobble-stones, but nothing else:

total = 1.0+0+0+0 = 1.0
total is greater than 0 so run the first “if”:
total is not greater than 1, so scalar = 1
result += texCob11 = texCob
result += texSand01 = result + 0 = result
result += texRock01 = result + 0 = result
result += texStone01 = result + 0 = result
total is not less than 1, so skip the second “if”

When there are cobblestones and sand, 0.7 to 0.5 respectively:

total = 0.7+0.5+0+0 = 1.2
total is greater than 0 so run the first “if”:
total is greater than 1, so scalar = 1.0/1.2 = 0.83
result += texCob0.70.83 = texCob0.58
result += texSand
0.50.83 = texCob0.58 + texSand0.42
result += texRock
01 = result + 0 = result
result += texStone
0*1 = result + 0 = result
total is not less than 1, so skip the second “if”

When there are cobblestones and sand, 0.2 to 0.5 respectively, so that grass should show through:
total = 0.2+0.5+0+0 = 0.7
total is greater than 0 so run the first “if”:
total is not greater than 1, so scalar = 1
result += texCob0.21 = texCob0.2
result += texSand
0.51 = texCob0.2 + texSand0.5
result += texRock
01 = result + 0 = result
result += texStone
01 = result + 0 = result
total is less than 1, so run the second “if”:
result += tex0
(1.0-0.7) = tex0*0.3

This looks pretty much correct to me, I believe.

Well what exactly do you want to happen when the mask has two values in one area? Blend in? Have earlier masks trump later masks?

Thaumaturge ninja’d in with a solution to fixing oversaturation while I was writing this. That method will probably work.

However, the easiest way to fix this is probably to just make your mask image having more than 1.0 in the sum of your channels an error which should be fixed in the image, not your shader. It would be faster on runtime, because the thing the shader will end up doing you could been have done in the image data beforehand. You could probably even write a python method that fixes the image like thaumaturge suggested one pixel at a time using some sort of image library. The code to do what thaumaturge suggested would look something like this

for pixel in image:
    total = r+g+b+a
    if total>1:
        scalar = 1.0/total
        r,g,b,a = scalar*r, scalar*g, scalar*b, scalar*a
    write pixel (this pixel, (r,g,b,a))

You could then write a dev shader that calculates the sum of all mask channels and replaces the final colour with (1.0,0,0,1.0) when this happens. Then you know that your channel image has an “error” that you need to fix whenever you see straight red.

Edit: Getting a bit ahead of myself back there…

Thaumaturge -> I think I got your code in cg, at least with no reported errors

float total = texA.r+texA.g+texA.b+texA.a ;    
     if (total > 0)
          float scalar = 1.0;
          if (total > 1.0)
               scalar = 1.0/total;
          result *= texCob*texA.r*scalar
                   + texSand*texA.g*scalar
                   + texRock*texA.b*scalar
                   + texStone*texA.a*scalar;
     if (total < 1.0)
          result *= tex0*(1.0-total);

But it still gives some problems (no sand, rock and stone blend well), can’t understand why.

Tober-> I think that is a good idea. I’ll be checking what PNMImage can do for me, either blending or just using the last mask would do.

Overall, I think that Tober’s suggestion of making oversaturation an error is perhaps not a bad idea.

As to the problem that you’re having with sections disappearing, I’m not sure of the result of chaining the additions after the multiplications; since multiplication seems to give the correct result, perhaps try bracketing the addition, like so:

result *= (texCob*texA.r*scalar
                   + texSand*texA.g*scalar
                   + texRock*texA.b*scalar
                   + texStone*texA.a*scalar);

Don’t use ifs in shaders. Seriously.

Personally, I’d say that it’s reasonable to assert that your alpha maps must already add up to a value of 1, and make sure that your content-creation tool errors out in this case. If you cannot rely on this, however, and want to blend equally even regardless of the alpha maps, then calculate the sum of all the factors and then divide each individual factor by this sum.

Thanks for all the tips.
After almost 3 hours of testing various solutions I ended up with more or less my first working solution.

result.rgba *=tex0*(1-texA.r)*(1-texA.g)*(1-texA.b)*(1-texA.a)

This is the only method that worked and seams universal.

I’m glad that you managed to come up with a working example. :slight_smile:

Looking at what you’ve done there, it looks like a good solution; if I read correctly it implies a heirarchy of sorts, with any one channel being set to 1 preventing subsequent channels having any effect, but if that’s producing results that you’re happy with then I see no great ill to that, I think. :slight_smile: