[SOLVED] Cardmaker and multiple TextureStages

Scripting Issues
1

I’ve got a texture captured from a webcam with people in it. I have used opencv to find the rectangle around a face in the image.
I want to:

  • take a portion of the image - defined by the rectangle
  • mask it out with a soft edged mask to frame the face
  • apply a helmet image over the top.
    It would look something like this (where original crowd image is much bigger)

So I am hoping to use the Cardmaker class and several texture stages.
But two of the textures are UV 0…1 but the original face is a tiny subset of the UV range.
So I need multiple UV sets on the Card.
But I can’t work out how to add them to a card object.
My mask image is:
My helmet image is:
(Any image will do for the face in a crowd image.)

I imagine my code might look something like this:
-but how do I get my UVs in there as well ?

  • alpha mask values not right yet.
   def make_card(self, x1,y1,x2,y2):
        texface = loader.loadTexture("models/crowd.jpg")
        texhelment = loader.loadTexture("models/helmet.png")
        texmask = loader.loadTexture("models/headmask.png")
        cm = CardMaker('head')
        cm.setFrame(-1,1,-1,1)
        self.facecard = render2d.attachNewNode(cm.generate(),2)
        self.facecard.setColor(0,0,0,1)
        self.facecard.setTransparency(TransparencyAttrib.MAlpha)
        #
        self.tsface = TextureStage('base')
        self.tsface.setMode(TextureStage.MReplace)
        self.facecard.setTexture(self.tsface, texface)
        #
        self.tsmask = TextureStage('mask')
        self.tsmask.setCombineAlpha(TextureStage.CMSubtract,
                                    TextureStage.CSPrevious,
                                    TextureStage.COSrcColor,
                                    TextureStage.CSTexture,
                                    TextureStage.COSrcAlpha)
        self.facecard.setTexture(self.tsmask, texmask)
        #
        self.tshelmet = TextureStage('helmet')
        self.tshelmet.setMode(TextureStage.MDecal)
        self.facecard.setTexture(self.tshelmet, texhelment)

Any ideas on how to make this work ?

2

I feel I’m getting close here.
Worked out I need to use setTexScale and setTexOffset on just the large original image.
(Headmask_alpha.png is a grayscale image)

If I comment out the self.tsmask lines below then it works except for the mask replacement.
Wondering if its something to do with original image not having an alpha to replace.
Also need to make sure I add my helmet mask to the result…

Any help ?

import direct.directbase.DirectStart
from direct.showbase.DirectObject import DirectObject
from panda3d.core import *
import sys

texface = loader.loadTexture("models/crowdA.png")
texhelmet = loader.loadTexture("models/helmet_01.00003.png")
texmask = loader.loadTexture("models/headmask-alpha.png")

class world(DirectObject):
    def __init__(self):
        self.accept('escape', sys.exit)
        rect = [414,180,482,268]
        self.make_card(rect[0],rect[1],rect[2],rect[3])

    def make_card(self, x1,y1,x2,y2):
        texmask.setFormat(texmask.FAlpha)
        width  = float(texface.getXSize())
        height = float(texface.getYSize())
        print width, height
        cm = CardMaker('head')
        cm.setFrame(x1/width*2-1, x2/width*2-1, y1/height*2-1,  y2/height*2-1)#(-1,1,-1,1) # L,R,B,T
        self.facecard = render2d.attachNewNode(cm.generate(),2)
        self.facecard.setColor(0,0,0,1)
        self.facecard.setTransparency(TransparencyAttrib.MAlpha)
        # OK and offsets working (except for padding)
        self.tsface = TextureStage('base')
        self.tsface.setMode(TextureStage.MReplace)
        self.facecard.setTexture(self.tsface, texface)
        hscale = (x2-x1) / width
        vscale = (y2-y1) / height
        xoffset = width/x1
        yoffset = height/y1
        self.facecard.setTexScale(self.tsface, hscale, vscale)
        self.facecard.setTexOffset(self.tsface, xoffset, yoffset)
        # not working - comment out to see partial
        self.tsmask = TextureStage('mask')
        self.tsmask.setCombineAlpha(TextureStage.CMReplace, 
                                    TextureStage.COSrcColor, TextureStage.COSrcAlpha)
        self.facecard.setTexture(self.tsmask, texmask)
        #self.facecard.setTexScale(self.tsmask, 1, 1)
        #self.facecard.setTexOffset(self.tsmask, 0, 0)
        # OK
        self.tshelmet = TextureStage('helmet')
        self.tshelmet.setMode(TextureStage.MDecal)
        self.facecard.setTexture(self.tshelmet, texhelmet)

w = world()
run()
3

You don’t have to have different uvs, you can just crop the original image with Panda’s image manipulation class, PNMImage. PNMImage can get images to and from a Texture object:

from panda3d.core import *
import direct.directbase.DirectStart

image = PNMImage()
image.read(Filename('test.png')) # or 'texture.store(image)' if you are getting the image from a Texture object

newimage = PNMImage(512, 512)
newimage.copySubImage(image, 0, 0, 50, 50, 512, 512) # from image, xto, yto, xfrom, yfrom, xsize, ysize

texture = Texture()
texture.load(newimage)

cm = CardMaker('cm')
cm.setFrame(-1,1,-1,1)
frame = render.attachNewNode(cm.generate())
frame.setTexture(texture)

run()

if your original image is that large, there’s no point in having it all along.

For the mask it’s easier to have white and transparency instead of white and black and use ts.setMode(TextureStage.MModulate).

Also in worse case you can just have multiple frames, with slightly different y offsets to avoid z-fighting, parented to a common node.

4

Thats terrifically useful thanks.
But it does open up another question (typical huh).

My source texture might have a region I wish to cut out that is say 59 pixels wide and 72 pixels high.
I want to avoid resampling it (if I can) but it doesn’t fit nicely into a binary pnm image size.
so that means masking or using UVs in the Cardmaker object again.

Or did I miss something ?

5

See here’s an image padded to 640x480 same as my camera.

and here’s a mask as a solid white with alpha mask
(kind of hiding on the white BG - but its on the left there)
and here’s a helmet.

My best guess at the code does this so far.
uvscale and offset helmet only including mask

but clearly a problem with the mask section. And it looks like a format problem…
But its not masking the original texture and its added grey dots…
test rectangles included below.

I really like the PNM approach and thanks for the MModulate tip. I tried it below and it worked better than anything I had already…

import direct.directbase.DirectStart
from direct.showbase.DirectObject import DirectObject
from panda3d.core import *
import sys

texface = loader.loadTexture("models/crowdA.jpg")
texhelmet = loader.loadTexture("models/helmet_01.00003.png")
texmask = loader.loadTexture("models/blankmask-2.png")

class world(DirectObject):
    def __init__(self):
        self.accept('escape', sys.exit)
##        self.rect = [19,19,48,48]   # upper Left
##        self.rect = [108,198,165, 273] # bloke left
##        self.rect = [299,189, 368, 246]  # midfield
##        self.rect = [483, 45, 498, 60]  # upper right
        self.rect = [418,192,477, 267]  #  Jimmy
##        self.rect = [12,437,41, 468]    # bottom left
##        self.rect = [601,415,630, 470]  # bottom right 
        self.make_card(self.rect[0],self.rect[1],self.rect[2],self.rect[3])

		
    def make_card(self, x1,y1,x2,y2):
        texmask.setFormat(texmask.FAlpha)
##        width  = float(texface.getXSize())
##        height = float(texface.getYSize())
        self.width = float(texface.getOrigFileXSize())
        self.height = float(texface.getOrigFileYSize())
        print self.width, self.height
        #print texface.getTexScale(), texface.getPadXSize(), texface.getPadYSize()
        self.cm = CardMaker('head')
        self.ll = x1/self.width*2-1
        self.rr = x2/self.width*2-1
        self.bb = y1/self.height*2-1
        self.tt = y2/self.height*2-1
        #self.cm.setFrame(self.ll,self.rr,self.bb,self.tt)#(-1,1,-1,1) # L,R,B,T
        self.cm.setFrame(-(x2-x1)/self.width,(x2-x1)/self.width,
                         -(y2-y1)/self.height, (y2-y1)/self.height)
        #self.cm.setFrame(-1,1,-1,1) # L,R,B,T
        self.facecard = render2d.attachNewNode(self.cm.generate(),2)
        self.facecard.setColor(0,0,0,1)
        self.facecard.setTransparency(TransparencyAttrib.MAlpha)
        # OK and offsets working
        self.tsface = TextureStage('base')
        self.tsface.setMode(TextureStage.MReplace)
        self.facecard.setTexture(self.tsface, texface)
        self.hscale = (x2-x1) / self.width
        self.vscale = (y2-y1) / self.height
        self.xoffset = x1/self.width
        self.yoffset = 1-y2/self.height
        #print self.xoffset, self.yoffset, self.hscale, self.vscale
        self.facecard.setTexScale(self.tsface, self.hscale, self.vscale)
        self.facecard.setTexOffset(self.tsface, self.xoffset, self.yoffset)
        # not working - comment out to see partial
##        self.tsmask = TextureStage('mask')
##        self.tsmask.setCombineAlpha(TextureStage.CMReplace, 
##                                    TextureStage.COSrcColor, TextureStage.COSrcAlpha)
##        self.facecard.setTexture(self.tsmask, texmask)
        #self.facecard.setTexScale(self.tsmask, 1, 1)
        #self.facecard.setTexOffset(self.tsmask, 0, 0)
        self.tsmask = TextureStage('mask')
        self.tsmask.setMode(TextureStage.Modulate)
        self.facecard.setTexture(self.tsmask, texmask)
        # OK
        self.tshelmet = TextureStage('helmet')
        self.tshelmet.setMode(TextureStage.MDecal)
        self.facecard.setTexture(self.tshelmet, texhelmet)
w = world()
run()

The reason I’m still doing the original texture is because I grab several faces from the one image. so I’m just keeping it around. But PNM would allow me to discard it and save space…

6

I don’t really understand the current problem. Maybe someone else can help you.

7

Oh I’m such an idot… sigh :frowning:
Right at the top of the function I set the mode to FAlpha. An early experiment that hung onto throttle me :frowning:
Taking that out I’m almost there.

Using MDecal at the end has the unfortunate effect of not adding the alphas. so I added a ts.combineAlpha but then lost my color :frowning:

I have a sneaking suspicion I need to to a combineRGB with a constant color but having difficulty setting a color on the texturestage.

I’m still asking for ideas ?

import direct.directbase.DirectStart
from direct.showbase.DirectObject import DirectObject
from panda3d.core import *
import sys

texface =   loader.loadTexture("models/crowdA.jpg")
texhelmet = loader.loadTexture("models/helmet_01.00003.png")
texmask =   loader.loadTexture("models/blankmask-2sq.png")

class world(DirectObject):
    def __init__(self):
        self.accept('escape', sys.exit)
        self.rect = [418,192,477, 267]  #  Jimmy
        self.make_card(self.rect[0],self.rect[1],self.rect[2],self.rect[3])

		
    def make_card(self, x1,y1,x2,y2):
        self.width = float(texface.getOrigFileXSize())
        self.height = float(texface.getOrigFileYSize())

        self.cm = CardMaker('head')
        self.cm.setFrame(-(x2-x1)/self.width,(x2-x1)/self.width,
                         -(y2-y1)/self.height, (y2-y1)/self.height)
        #self.cm.setFrame(-1,1,-1,1) # L,R,B,T
        self.facecard = render2d.attachNewNode(self.cm.generate(),2)
        self.facecard.setColor(0,0,0,1)
        self.facecard.setTransparency(TransparencyAttrib.MAlpha)
        # OK offsets working
        self.tsface = TextureStage('base')
        self.tsface.setMode(TextureStage.MReplace)
        self.facecard.setTexture(self.tsface, texface,1)
        self.hscale = (x2-x1) / self.width
        self.vscale = (y2-y1) / self.height
        self.xoffset = x1/self.width
        self.yoffset = 1-y2/self.height

        self.facecard.setTexScale(self.tsface, self.hscale, self.vscale)
        self.facecard.setTexOffset(self.tsface, self.xoffset, self.yoffset)
        #
        self.tsmask = TextureStage('mask')
        self.tsmask.setMode(TextureStage.MModulate)
        self.facecard.setTexture(self.tsmask, texmask,2)
        # OK
        self.tshelmet = TextureStage('helmet')
        self.tshelmet.setMode(TextureStage.MDecal)
        #self.tshelmask.setColor((0,0,0,1))
##        self.tshelmet.setCombineRgb(TextureStage.CMModulate,
##                                    TextureStage.CSPrevious, TextureStage.COSrcColor,
##                                    TextureStage.CSTexture, TextureStage.COOneMinusSrcAlpha)
        self.tshelmet.setCombineAlpha(TextureStage.CMAdd,
                                      TextureStage.CSPrevious, TextureStage.COSrcAlpha,
                                      TextureStage.CSTexture, TextureStage.COSrcAlpha)
        self.facecard.setTexture(self.tshelmet, texhelmet,3)

w = world()
run()
8

OK. worked it out.

I needed to add a fourth stage (maximum capability of my graphics card - so glad I didn’t need more). In case you need to know its:

base.win.getGsg().getMaxTextureStages()

Maybe there is a way to do it with less. I suspect there is.

My solution: (cleaned up)

import direct.directbase.DirectStart
from direct.showbase.DirectObject import DirectObject
from panda3d.core import *
import sys

facetex   = loader.loadTexture("models/crowdA.jpg")
helmettex = loader.loadTexture("models/helmet_01.00003.png")
mask      = loader.loadTexture("models/blankmask-2sq.png")

class world(DirectObject):

    def __init__(self):
        self.accept('escape', sys.exit)
        self.rect = [418,180,477, 267]  #  Jimmy 
        self.make_card(render2d, self.rect[0],self.rect[1],self.rect[2],self.rect[3],
                       facetex, mask, helmettex)

		
    def make_card(self, parent, x1,y1,x2,y2, basetex, texmask, texoverlay):
        """ Make a new card with:
            - the rectangle cut from basetex (RGB image)
            - texmask cutting into it (whiteFG, mask for BG)
            - texoverlay applied on top
            Where texmask and texoverlay are RGBA images with active masks
        """
            
        width = float(basetex.getOrigFileXSize())
        height = float(basetex.getOrigFileYSize())
        
        cm = CardMaker('head')
        # Use rect's aspect ratio
        cm.setFrame(-(x2-x1) / width,(x2-x1) / width,
                    -(y2-y1) / height, (y2-y1) / height)
        self.facecard = parent.attachNewNode(cm.generate())
        self.facecard.setTransparency(TransparencyAttrib.MAlpha)
        
        # Apply texture to new Card. calc UV offset and scale to fit
        tsface = TextureStage('base')
        tsface.setMode(TextureStage.MReplace)
        self.facecard.setTexture(tsface, basetex, 1)
        hscale = (x2-x1) / width
        vscale = (y2-y1) / height
        xoffset = x1 / width
        yoffset = 1-y2 / height
        self.facecard.setTexScale(tsface, hscale, vscale)
        self.facecard.setTexOffset(tsface, xoffset, yoffset)
        
        # mask out BG
        tsmask = TextureStage('mask')
        tsmask.setMode(TextureStage.MModulate)
        self.facecard.setTexture(tsmask, texmask, 2)
        
        # Add Overlay on top but ignore its texture
        tshelmet = TextureStage('helmet')
        tshelmet.setColor(VBase4(1,1,1,1))
        tshelmet.setCombineAlpha(TextureStage.CMAdd,
                                 TextureStage.CSPrevious, TextureStage.COSrcAlpha,
                                 TextureStage.CSTexture, TextureStage.COSrcAlpha)
        tshelmet.setCombineRgb(TextureStage.CMModulate,
                               TextureStage.CSPrevious, TextureStage.COSrcColor,
                               TextureStage.CSTexture, TextureStage.COOneMinusSrcAlpha)
        self.facecard.setTexture(tshelmet, texoverlay, 3)
        
        # add Overlay texture back on
        tshelmask = TextureStage('helmask')
        tshelmask.setMode(TextureStage.MDecal)
        self.facecard.setTexture(tshelmask, texoverlay, 4)
        

w = world()
run()

Side note - if the texmask is not quite white then it will tint the base texture.
Cheers…

9

If you just need a single output texture (don’t need to transform each separate texture later), then just bake them into one. If you are using the fixed function, instead of custom shaders or the shader generator, some old GPUs won’t work as they have a limit of 2 TextureStages.

PNMImage’s blend() and blendSubImage() methods seem to allow you to do just that.
For the white and transparent texture, you can even just multiply the old PNMImage with it (pnmimage * pnmimage).

If that didn’t exist you could set your images as brushes of PNMPainter and apply them like that to a PNMImage.

There’s also the MultitexReducer class which bakes TextureStages to one, haven’t tried that one though.

If you don’t want to lose control of the final TextureStage (hats), then you can always only merge the previous ones.

And again unless you’re applying the texture to some uneven surface, you can just create multiple cards and parent them to a common node, changing their y pos a little to avoid z-fighting.

parent = render.attachNewNode('parent')
for i in myframes:
    i.reparentTo(parent)
    i.setY(parent, -0.1*myframes.index(i))

PIL can also do image manipulations like this, pythonware.com/products/pil/