Threading using Python

With Panda3D 1.6.2 I could get about 4,000 threads running in parallel under simple threading. All I had to do was to set the thread-stack-size to 65536.

With 1.7.0 I cannot get more than a handful of threads running using the same code, and the program exits silently (aka crashes quietly) at the touch of a, well, thread.

I’ll try to sort out a simple example that demonstrates the problem.

I have my reasons for wanting to be able to run a lot of threads, and seemed to be able to do that with 1.6.2. If there is some setting I can tweak pls let me know, otherwise I will work on getting a simple example sorted out for this issue.

Cheers

Hmm, I’m not aware of anything that changed substantially between 1.6.2 and 1.7.0 with regards to limitations on the number of threads. However, note that in 1.7.1 and later, Panda will try to use OS constructs to create its simple threads, which may also inherently restrict the maximum number of threads that may be created.

David

My mistake - I had been experimenting with threading2. Turns out that what I am doing works fine with threading, but runs out of memory (or worse) with threading2. Nothing to do with Panda3D version at all.

Hi,

I'm interesting in this topic, too.

I try to find a way to execute spirite's logic function 500 times each second.

I use taskMgr.setupTaskChain to create another task chain by set numThreads = 1, and add my spirite's logic function into this task chain. It works to execute function more often rendering. 

The next step is to fix the execution rate.

I found that taskMgr.doMethodLater can do task later. When I set the delayTime less then current frame rate(like 0.005), I only got the same execution rate equal to current frame rate. My question is, how can I do to fix the task's execution rate more then my current frame rate?

regards

If your task is added to a separate task chain, it can execute more often than once per frame. You have to be sure not to set frameSync = True, leave it the default, False.

Still, you should understand that this isn’t a way to create CPU cycles out of nothing. Unless you have compiled Panda yourself to use true threads and you have multiple CPU’s on your machine, then any time you spend in your sprite’s logic function is still deducted from your overall frame rate (because the CPU can only do one thing at a time, so when you’re calculating sprites, you’re not drawing the frame).

Also, Panda’s task system is not intended to be a real-time scheduler, so it doesn’t provide a way to guarantee that a certain task executes a certain number of times per second.

David

Here is some simple code that demonstrates the Assertion error that I have been getting. I have been working for two weeks to get it down to under 400 lines (the error is not always easy to invoke, I suspect it is a dangling pointer or other sort of pointer problem and can therefore appear to be intermittent -the following code seems to cause the error consistently)

# This code demonstrates the assertion error:
# Assertion failed: _states->find(this) == _saved_entry at line 1944 of c:\panda3d-1.7.0\panda\src\pgraph\transformState.cxx
# Assertion failed: _saved_entry == _states->end() at line 108 of c:\panda3d-1.7.0\panda\src\pgraph\transformState.cxx
# Author: Chris Hilder, email: cj.hilder@clear.net.nz, 24 July 2010

from direct.directbase import DirectStart
from direct.showbase.DirectObject import DirectObject
from direct.gui.DirectGui import *
from direct.interval.IntervalGlobal import *
from direct.actor.Actor import Actor
from direct.stdpy import threading
from direct.showbase.PythonUtil import Queue  

from pandac.PandaModules import lookAt
from pandac.PandaModules import Thread  # necessary for Thread.sleep() which does not exist in threading
from pandac.PandaModules import GeomVertexFormat, GeomVertexData
from pandac.PandaModules import Geom, GeomTriangles, GeomLines, GeomVertexWriter
from pandac.PandaModules import Texture, GeomNode, TextureStage, TexGenAttrib
from pandac.PandaModules import TextFont
from pandac.PandaModules import PerspectiveLens
from pandac.PandaModules import CardMaker
from pandac.PandaModules import Light, Spotlight
from pandac.PandaModules import TextNode
from pandac.PandaModules import Vec3, Vec4, Point3, Point4, Mat3, Mat4, TransformState
from pandac.PandaModules import ConfigVariableString

#-----------------------------------------------------------------------------
# A couple of drawing functions that create new geometry...

def _drawTri(A, B, C, colourA, colourB, colourC, alpha):

    x1, y1, z1 = A
    x2, y2, z2 = B
    x3, y3, z3 = C
    r1, g1, b1 = colourA
    r2, g2, b2 = colourB
    r3, g3, b3 = colourC
    
    trianglePrimitive = GeomTriangles(Geom.UHStatic)
    format = GeomVertexFormat.getV3c4()  # standard vertex format with a 4-component color and a 3-component vertex position
    vdata = GeomVertexData('triangle', format, Geom.UHStatic)
    vertex = GeomVertexWriter(vdata, 'vertex')
    color = GeomVertexWriter(vdata, 'color')
            
    vertex.addData3f(x1, y1, z1)
    vertex.addData3f(x2, y2, z2)
    vertex.addData3f(x3, y3, z3)
    color.addData4f((r1, g1, b1, alpha))
    color.addData4f((r2, g2, b2, alpha))
    color.addData4f((r3, g3, b3, alpha))
          
    trianglePrimitive .addVertex(0)
    trianglePrimitive .addVertex(1)
    trianglePrimitive .addVertex(2)
    trianglePrimitive .closePrimitive()
    triangle = Geom(vdata)
    triangle.addPrimitive(trianglePrimitive )

    snode = GeomNode('triangle')
    snode.addGeom(triangle)
    return snode

def _drawLine(start, end, startColour, endColour):

    x1, y1, z1 = start
    x2, y2, z2 = end

    linePrimitive = GeomLines(Geom.UHStatic)
    format = GeomVertexFormat.getV3c4()  # standard vertex format with a 4-component color and a 3-component vertex position
    vdata = GeomVertexData('line', format, Geom.UHStatic)
    vertex = GeomVertexWriter(vdata, 'vertex')
    color = GeomVertexWriter(vdata, 'color')
            
    vertex.addData3f(x1, y1, z1)
    vertex.addData3f(x2, y2, z2)
            
    color.addData4f(startColour)
    color.addData4f(endColour)
            
    linePrimitive.addVertex(0)
    linePrimitive.addVertex(1)
    linePrimitive.closePrimitive()
    line = Geom(vdata)
    line.addPrimitive(linePrimitive)

    snode = GeomNode('line')
    snode.addGeom(line)
    return snode
#-----------------------------------------------------------------------------

class Turtle(object):
    """ An implementation of turtlegraphics. """
    
    def __init__(self, actor = "defaultActor", font = "defaultFont"):

        if font == "defaultFont":
            font = defaultFont

        self.colourA = (0.0, 0.0, 0.0)
        self.colourB = (0.0, 0.0, 0.0)
        self.colourC = (0.0, 0.0, 0.0)
        self.alpha   = 1.0
        
        self.pointA = Point3(0.0, 0.0, 0.0)
        self.pointB = Point3(0.0, 0.0, 0.0)
        self.pointC = Point3(0.0, 0.0, 0.0)
        self.texture = None
        
        self.drawingNode       = render.attachNewNode("Drawing")
        # This node has the drawing attached as it is generated by Turtle moves
        
        self.locationNode      = render.attachNewNode("Location")                 
        # This node locates the Turtle in rendering space.

        self.adjustmentNode    = self.locationNode.attachNewNode("Adjustment")    
        # This node is used for calculating Turtle moves

        self.penDown()
        self.textFont = font
        
    def __del__(self):
        self.drawingNode.removeNode()
        self.locationNode.removeNode()

    def penUp(self):
        self.penIsDown = False
       
    def penDown(self):
        self.penIsDown = True
           
    def clear(self):
        self.locationNode.setTransform(self.locationNode.getTransform().makeIdentity())
        
        self.drawingNode.removeChildren()
        ########################################
        # The above line is the trigger of the error.
        # Replacing it with stash, detatchNode, or reparentTo sometimes stops the error
        # from happening, but not always. After numerous experiments
        # I conclude that any change to the scene graph,
        # such as attaching a new geom node, is sufficient
        # to trigger the error, provided the right situation
        # has been set up. With the present geometry
        # the removeChildren() is a perfect trigger.
        ########################################

    def setPos(self, x=None, y=None, z=None):
        """Can take a tuple of 1, 2, or 3 coordinates, or separate parameters (1, 2, or 3 of them).
        The first coordinate is x, the second y, and the third z. Coordinates left out remain unchanged."""
        if isinstance(x, Vec3):
            pos = Point3(x)
        else:
            pos = x
        if isinstance(pos, tuple) or isinstance(pos, Point3):

            if len(pos) == 2: 
                x, y = pos
                z = self.locationNode.getZ()
                pos = (x, y, z)
            if len(pos) == 1:
                x, y, z = self.locationNode.getPos()
                x = pos[0]
                pos = (x, y, z)
        else:
            if (not x is None) and (not y is None) and (not z is None):
                pos = (pos, y, z)
            else:
                x1, y1, z1 = self.locationNode.getPos()
                if not x is None:
                    x1 = float(x)
                if not y is None:
                    y1 = float(y)
                if not z is None:
                    z1 = float(z)
                pos = (x1, y1, z1)

        if self.penIsDown:
            c1r, c1g, c1b = self.colourA
            c2r, c2g, c2b = self.colourB
            node =_drawLine(self.locationNode.getPos(render), pos, 
                           Vec4(c1r, c1g, c1b, self.alpha), Vec4(c2r, c2g, c2b, self.alpha))
            self.drawingNode.attachNewNode(node)
        self.locationNode.setPos(pos)
    
    def forward(self, steps):
        steps = float(steps)

        if self.penIsDown:
            c1r, c1g, c1b = self.colourA
            c2r, c2g, c2b = self.colourB
            node =_drawLine(Vec3(0.0, 0.0, 0.0), Vec3(0.0, steps, 0.0), 
                           Vec4(c1r, c1g, c1b, self.alpha), Vec4(c2r, c2g, c2b, self.alpha))
            new = self.drawingNode.attachNewNode(node)
            new.setPos(self.locationNode.getPos(render))
            new.setHpr(self.locationNode.getHpr(render))
        self.adjustmentNode.setPos(0.0, steps, 0.0)
        self.locationNode.setPos(self.adjustmentNode.getPos(render))
        self.adjustmentNode.setPos(0.0, 0.0, 0.0)

    def back(self, steps):
        self.forward(-steps)

    def _rotate(self, angle, hpr):
        self.adjustmentNode.setHpr(hpr)
        self.locationNode.setHpr(self.adjustmentNode.getHpr(render))
        self.adjustmentNode.setHpr(0.0, 0.0, 0.0)
        
    def left(self, angle):
        angle = float(angle)
        self._rotate(angle, (angle, 0.0, 0.0))
                
    def right(self, angle):
        self.left(-angle)
        
    def up(self, angle):
        angle = float(angle)
        self._rotate(angle, (0.0, angle, 0.0))
        
    def down(self, angle):
        self.up(-angle)
        
    def rollRight(self, angle):
        angle = float(angle)
        self._rotate(angle, (0.0, 0.0, angle))
        
    def rollLeft(self, angle):
        self.rollRight(-angle)

    def setFont(self, font):
        self.textFont = font
        
    def write(self, text, font = None):
        if font is None:
            font = self.textFont
        if font is None:
            print "Please tell me which font to use (or set a default font)."
        else:
            textNodePath = self.drawingNode.attachNewNode("Text Node Path")
            textNodePath.setPos(self.locationNode.getPos())
            textNodePath.setHpr(self.locationNode.getHpr())
            textNodePath.setScale(self.adjustmentNode.getScale())

            newNode = TextNode("Text")
            newNode.setFont(font.font)
            newNode.setText(text)

            x, y, z = self.colourA
            w = self.alpha
            newNode.setTextColor(x, y, z, w)

            newNodePath = textNodePath.attachNewNode(newNode.generate())
            newNodePath.setPos(-2.0, 0.0, 0.0)
            newNodePath.setHpr(90.0, -90.0, 0.0)
            newNodePath.setScale(font.scale)
            newNodePath.setTwoSided(True)

    def setA(self, x = None, y = None, z = None):
        self.pointA = self._convertToPoint3(x, y, z)

    def setB(self, x = None, y = None, z = None):
        self.pointB = self._convertToPoint3(x, y, z)

    def setC(self, x = None, y = None, z = None):
        self.pointC = self._convertToPoint3(x, y, z)

    def setTexture(self, texture):
        self.texture = texture

    def triangle(self, A = None, B = None, C = None):
        if A is None:
            A = self.pointA
        if B is None:
            B = self.pointB
        if C is None:
            C = self.pointC
        if A == B or B == C or C == A:
            return
        if self.texture is None:
            tri = _drawTri(A, B, C, self.colourA, self.colourB, self.colourC, self.alpha)
        else:
            white = (1.0, 1.0, 1.0)
            tri = _drawTri(A, B, C, white, white, white, self.alpha)
            
        newNode = self.drawingNode.attachNewNode(tri)
        newNode.setTwoSided(True)
        if self.texture is not None:

            newNode.setTexGen(TextureStage.getDefault(), TexGenAttrib.MWorldPosition)
            newNode.setTexProjector(TextureStage.getDefault(), render, newNode);
            newNode.setTexture(self.texture, 1)
            newNode.setTexScale(TextureStage.getDefault(),0.0100,0.0100)
             
    def _convertToPoint3(self, x = None, y = None, z = None):
        if isinstance(x, Point3):
            return x
        if isinstance(x, Vec3):
            return Point3(x)
        if isinstance(x, tuple):

            if len(x) == 2: 
                x1, y1 = x
                z1 = self.locationNode.getZ()
                return Point3(x1, y1, z1)
            if len(x) == 1:
                x1, y1, z1 = self.locationNode.getPos()
                x1 = x[0]
                return Point3(x1, y1, z1)
        else:
            if (not x is None) and (not y is None) and (not z is None):
                return Point3(x, y, z)
            else:
                x1, y1, z1 = self.locationNode.getPos()
                if not x is None:
                    x1 = float(x)
                if not y is None:
                    y1 = float(y)
                if not z is None:
                    z1 = float(z)
                return Point3(x1, y1, z1)

    def toss(self, speed = 1):
        self.spinEffect = self.drawingNode.hprInterval(speed, Point3(self.drawingNode.getH()+360,self.drawingNode.getP()+360,self.drawingNode.getR()+360))
        self.spinEffect.start()

#-----------------------------------------------------------------------------

# Some code to exercise the Turtle by generating some geometry.
# It generates some triangles with applied texture, some lines, and some text nodes.
# The exact make-up of the geometry does not seem to affect the occurrance of the error,
# but this particular example seems to cause the error reasonably reliably...

def tris():
    for i  in range(18):
        tri()
        x.right(15)
        x.up(15)

def tri():
    x.setTexture(blobby)
    x.forward(100)
    x.right(120)
    x.setA()
    x.write('Bug')
    x.forward(100)
    x.right(120)
    x.setB()
    x.write('Bug')
    x.forward(100)
    x.right(120)
    x.setC()
    x.write('Bugger')
    x.triangle()

def triThing():
    x.clear()
    x.penDown()

    tris()
    x.toss(1)

def waitFor(seq):
    Thread.sleep(0.04)
    while seq.isPlaying():
        Thread.sleep(0.04)

def causeBugToShow():
    global x
    x = Turtle(None)
    triThing()
    waitFor(x.spinEffect)
    x.clear()

#-----------------------------------------------------------------------------

class CicadaFont(object):
    """ This class is just a way of loading a font and associating a certain scale number with it. """
    def __init__(self, fontFileName, fontScale = 24):
        try:
            self.font = loader.loadFont(fontFileName)
        except:
            print "Unable to load font:", fontFileName
        self.scale = fontScale

class _ExecThread(threading.Thread):
    """ The main worker thread, from which all commands are executed using eval(). """
    def run(self):
        while True:  
            while not _commandLineQueue.isEmpty():
                command = _commandLineQueue.pop()
                eval(command, globals())
                Thread.sleep(0.04) 
            Thread.sleep(0.04) 

_commandLineQueue = Queue()
_thread = _ExecThread()
_thread.start()

blobby = loader.loadTexture('models/maps/noise.rgb')
defaultFont = CicadaFont('models/cmtt12.egg', fontScale = 16)
base.disableMouse()
base.camera.setPos(100.0, 100.0, 1000.0)
base.camera.lookAt(100.0, 100.0, 0.0)

_commandLineQueue.push('causeBugToShow()')
run()

Hey David, are you able to replicate this error at your end? -Chris

Sorry, I’ve been out of town for a few days. I’ll take a stab at this soon. :slight_smile:

David

Thanks David, will look forward to hearing how you get on. Just a small update: I’m pretty sure the error doesn’t occur when new geometry is added to the scene graph, but only when existing geometry that has been animated with an interval has its place in the scene graph changed i.e. remove, stash, detatch, or reparent.

Cheers

A further update: I rewrote my code so it doesn’t use intervals, sequences, or parallels at all. Still getting the error message. So it is not caused by intervals after all. Looks like it might arise simply from manipulating the scene graph from within a thread, with bigger manipulations more likely to cause the error.

Hmm, I’ve been trying to run your code to reproduce the error, and it hasn’t been crashing for me at all. But I am running with a very recent cvs build. Are you using the stock 1.7.0 build for Windows? Perhaps this is a problem that has been fixed on the trunk. Try installing the buildbot release to see if you still get the problem there.

David

Thanks for giving my code a whirl. Good news. It looks like the error has been fixed. I have installed Panda3D-2010.07.29-86 (version 1.7.1) on two different machines, and on both machines the error no longer occurrs with my test code (posted in my earlier post).

I have not been able to run a full suite of tests because 1.7.1 breaks my main program (it runs ok under 1.7.0). It looks like I will need to use 1.7.1 so I am going to have to work out what needs to be done to get my program running again. Can you point me towards a list of changes between 1.7.0 stock release and 1.7.1 bleeding edge as of now? That will help me track down the cause of my latest problem and decide what to do about it. :slight_smile:

There aren’t supposed to be any API changes between a .0 release and a .1 release. The minor releases are generally intended to be bugfix-only. That is, if it works in 1.7.0, it’s supposed to work in 1.7.1, unless you were relying on a particular buggy behavior.

So, can you tell us more about the way in which your application fails in 1.7.1?

David

Note that that may not apply for the buildbot releases, as the buildbot builds from trunk, not from the 1.7 branch.

So the buildbot release is incorrectly carrying the 1.7.1 version number. David, do you think it would be a good idea to make a specialised interface in PandaSystem that can store information about the daily build?

Ah yes, hmm, the buildbot should have a separate version number that’s not in the 1.7.x series; perhaps it should be version 1.8.0c, where the ‘c’ is Panda’s convention to indicate a version-in-progress. The “1.8” part is still a little misleading, but any other convention is misleading in other ways.

David

I have been trying to work out what part of my code has caused the problem with the latest buildbot version. As I try different combinations of things the problems come and go. This is a very similar situation to what I was getting under 1.7.0 and 1.6.2. I suspect the cause is actually the same bug, and that it has not actually been fixed between 1.7.0 and 1.7.1. In its latest incarnation it strikes on calls to wxPython; earlier it was striking the Panda code. I have prior experience of memory corruption bugs in C, and this feels very much like one of those. It will be very consistent if you use the same code, but will come and go like a chimera if you change the code making you think the part you changed is somehow involved in the bug when it is nothing more than an innocent bystander. It took me nearly two weeks to isolate a small bit of code (400 lines) that consistently generates the error on my machine, but I have since tried it on another machine on which it runs perfectly. This makes it almost impossible to develop and send code snippets that reproduce the problem reliably. I think I would have to send 1,500 lines of code for you to play around with until you found a combination of moves that consistently generate the problem on your machine. Do you really want to do that?

For now there are a couple of questions that might help me move forward:

  1. Are there any known issues with Panda3D and wxPython that have emerged since version 1.7.0?

  2. Is there anyone else who is doing what I am doing and generating/manipulating lots of Panda3D geometry from within a thread?

  3. Would native (true) python threads be safe to use with Panda3D provided I used a suitable locking regime (e.g. aquire a lock before any Panda3D operation, and have Panda itself aquire a lock which is only released in a specific task, and then acquired again before exiting that task)?

  4. Is it possible for the BuildBot to generate a version using true Panda3D threads, as opposed to simple threading? Or do I have to do a build myself if I want to experiment with this option?

I’m leaning toward the option in question 3 at the moment because I feel that I am thrashing about futilely with my present course of action.

Any comments and replies appreciated.

Chris

These are good questions.

  1. I don’t know of any such issues, but I haven’t been heavily exercising wxPython myself. However, the Level Editor teams (both the team here in California, and the team in the ETC) have been using it fairly heavily, and I haven’t heard complaints.

  2. I think you’re blazing new territory here. However, we do use asynchronous loading of models in the Pirates game, and this means generation of geometry and textures from a BAM file in a sub-thread. We haven’t had trouble with it so far. However, I wouldn’t at all be surprised to find something not-quite-thread-safe in some part of the geometry-generation pipeline, maybe a part that doesn’t get used when only loading models from BAM files.

  3. Yes, but it is difficult to do this with reference-counted primitives. You would have to acquire that lock around each dereference operation; and these often happen implicitly. A better approach would probably be to build Panda with true threads instead, which leads to the next question:

  4. I think it is best to build Panda3D yourself. It is not really that difficult to do this. I’m not sure this will solve your problem, though; if there is a race condition within Panda itself, using true threads will only aggravate it.

David

I am, and while I’m not getting the performance I’d like, I’m not having any stability issues.

I’m spawning a regular old python threading.Thread, doing lots of computing to produce a procedural 3d model on a GeomNode. I’m very careful to isolate that thread - I pass the parameters it needs in on one (python standard, thread-safe) Queue, and pass back the finished model on another Queue.

My complaint with this scheme right now is this: if I run the model construction code on the main thread, it blocks for about 3-6 seconds to build the model, using ~100% of one core. I put it on a separate thread while displaying a fairly simple ‘splash screen’ - just a few text nodes - so there would be an attractive visual while waiting for the models to be built.

What I expected was for the main thread processing and rendering to use a small fraction of the CPU while the rest of a core chewed on the model building. Unfortunately what happens is the app uses ~25% of one core and the model building takes 15+ seconds.

If I use a threaded task chain instead, the app becomes unresponsive while model building is in progress, so that’s not really an option. (SIMPLE_THREADS build, obviously, and I’m not really looking forward to rebuilding panda.) Ditto if I use either panda threading or threading2. As an aside, I find it pretty ridiculous that the documentation pushes threading2 as having “the same semantics as python threads” when it requires explicit yield calls to do anything remotely useful.

So my choices right now seem to be:

  • Leave my CPU 75% idle and take 15-20 seconds to build a model
  • Build a non-SIMPLE_THREADs version of Panda
  • Sprinkle cooperative-multitasking yield-turds throughout my model builder code and party like it’s 1999
  • Use the python multiprocessing module and take advantage of the second core…

So, side question raised by that last option - Panda needs me to run ppython == Mac OS preinstalled python 2.5, while multiprocessing was introduced in python 2.6. Is Panda 1.7 on non-Apple Python 2.6 on Snow Leopard a thing that can happen?

Not much you can do really with threads… They’re perty much only used to get around blocking IO at the cost of lossing 10% + 50% from the second thread not getting all the tricks. Best way to break it up is to have the heavy loading on your main and do the light on the second thread. This way, when your light thread is done, it can give it’s power back to the main thread. (Note: this has to do with the cpu and python, not so much panada.)

The only other way is to use the multiprocessing module. Last time I used it… it had some really big issues with panad3d… so maybe option 1 would be the best way. Also note… on windows… the multiprocessing module is really heavy, so you take on even more overhead. I think the main reason is cause there arn’t a lot of forks in the proccess (so maybe a dev can take a look at it and remake a new one) this way you have less overhead I think making each proccess just as fast.

It’s taken me ages to get this sorted, but by using the latest snapshot build (Panda3D-2010.08.20-108) I am no longer getting any assertion errors, which were the big problem I asked about above. It looks as if the bug that caused these has been fixed. All my test code now runs without error.

Initially I was getting problems with wxPython under the latest snapshot. I dicovered I was calling wxPython funtions from different tasks and/or threads. This was working fine in version 1.7.0, so there must be something quite different in 1.7.1 causing it to give major problems. Problem solved by passing messages and calling wxPython always from the same task, which is the recommended practice according to wxPython documentation. Further problems have arisen with wxPython but I have posted elsewhere because they are nothing to do with threads.

Thank for the generous help in this thread, and sorry I can’t offer any comments about performance issues under threads - my code still isn’t working well enough to get much idea of overall performance!