C++ Module using Python-side Messenger -> Deadlock

Scripting Issues
1

Hi all,

hope, this is the right section to post. I am struggling with the Messenger on Panda3D’s Python side.

This is what I want to have: A C++ Extension Module which takes a Python Callback to eventually feed the Messenger. But it always ends up with a deadlock, sooner or later.

Full runnable snippet:

Script

import direct.directbase.DirectStart
from direct.showbase.DirectObject import DirectObject


from EventGenerator import EventGenerator

			
class EventReceiver(DirectObject):
	def __init__(self):
		DirectObject.__init__(self)
		
		self.accept('escape', self.release)

		self.generator = EventGenerator(self, 'getEvent')
		
	
	def getEvent(self, event):
		print event
		messenger.send(event)
		#messenger.send(event, taskChain='myTC')
		
		
		
	def release(self):
		self.generator.stop()
		del self.generator
		exit()
		


EventReceiver()	
run()

Simple C++ Module

//EventGenerator.cpp
#include <cstdlib>	//for rand

#include <boost/python.hpp>
#include <boost/thread.hpp>
#include <boost/utility.hpp>


class EventGenerator : boost::noncopyable
{
	public:
		EventGenerator(PyObject*, const char* pyMethod);
		~EventGenerator();

		void generate() const;
		void stop();


	private:
		PyObject* instance;
		const char* method;

		bool requestStop;
		boost::thread t;
};


#pragma warning(push)
#pragma warning(disable: 4355)	//for 'this' in initializer list. There is no need to warn since "generate" does not depend on object construction completion.

EventGenerator::EventGenerator(PyObject* po, const char* pyMethod)
	: instance(po), method(pyMethod), requestStop(false), t(&EventGenerator::generate, this)
{}

#pragma warning(pop)


EventGenerator::~EventGenerator()
{
	t.join();
}


void EventGenerator::generate() const
{
	std::srand(std::time(0));
	while (!requestStop)
	{
		if((std::rand()%10) < 3)	//this is just a replacement for doing some fancy stuff in fact
		{
			PyGILState_STATE gstate = PyGILState_Ensure();
			boost::python::call_method<void>(instance, method, "hello from c++");
			PyGILState_Release(gstate);
		}
	}
}


void EventGenerator::stop()
{
	requestStop = true;
}



BOOST_PYTHON_MODULE(EventGenerator)
{
	using namespace boost::python;
	
	class_<EventGenerator, boost::noncopyable>("EventGenerator", init<PyObject*, const char*>())
		.def("stop", &EventGenerator::stop);
}

Any idea? It seems that the locking requests from python and c++ side eventually interfer. But why?

OS: Windows 7 64 Bit
Visual Studio 2010
Wrapper: Boost 1.44

Panda3D 1.7

Yours,
rahdev

2

There are some complex rules you have to follow to set the locks correctly when calling into Python from Panda3D C++ code. Panda3D has code to deal with this in Thread::call_python_func(). I don’t know whether it plays well with boost, though.

David

3

Thanks for your immediate reply, David. Thought, the PyGILState_Ensure/Release pair should be enough. As suggested, I have taken a look at Thread::call_python_func and copied some code, but, unfortunately it still does not work.

void EventGenerator::generate() const
{
	std::srand(std::time(0));
	std::vector<PyThreadState *> thread_states;
	while (!requestStop)
	{
		if((std::rand()%10) < 3)	//this is just a replacement for doing some fancy stuff in fact
		{	

			PyThreadState *orig_thread_state = PyThreadState_Get();
			PyInterpreterState *istate = orig_thread_state->interp;
			PyThreadState *new_thread_state;
			if (thread_states.empty()) {
			  new_thread_state = PyThreadState_New(istate);
			} else {
			  new_thread_state = thread_states.back();
			  thread_states.pop_back();
			}
			PyThreadState_Swap(new_thread_state);
			
			
			boost::python::call_method<void>(instance, method, "hello from c++");
				
			
			PyThreadState *state = PyThreadState_Swap(orig_thread_state);
			 thread_states.push_back(new_thread_state);

		}
	}
}

Panda3D says:

Through a web search I found another snippet:

void EventGenerator::generate() const
{
	std::srand(std::time(0));
	
	while (!requestStop)
	{
		if((std::rand()%10) < 3)	//this is just a replacement for doing some fancy stuff in fact
		{
			if (!PyEval_ThreadsInitialized()) 
			{
				PyEval_InitThreads();
			// PyEval_InitThreads() acquires the GIL on my behalf but
			// I don't want it at the moment.
				PyEval_ReleaseLock();
			}


			PyGILState_STATE gstate;
			PyThreadState *main_thread;
			PyThreadState *callback_thread;


			// PyGILState_Ensure() implicitly acquires the GIL so I don't need
			// to call PyEval_AcquireLock().
			gstate = PyGILState_Ensure();

			// Get the current thread state so that I have an interpreter to
			// which to point.
			main_thread = PyThreadState_Get();

			// Create a new Python thread for the callback.
			callback_thread = PyThreadState_New(main_thread->interp);

			// Make the callback thread current.
			PyThreadState_Swap(callback_thread);

			// Perform the callback.
			
			boost::python::call_method<void>(instance, method, "hello from c++");

			// Clean up my internal pointers
	

			// Now unwind the Python thread/GIL stuff above
			PyThreadState_Swap(main_thread);

			PyThreadState_Clear(callback_thread);

			PyThreadState_Delete(callback_thread);

			// PyGILState_Ensure() acquires the lock, but does
			// release it? The documentation doesn't say, but it seems like it does.
			
			PyGILState_Release(gstate);

		}
	}
}

which also results in deadlock, sooner or later, seems to be random time.

Regarding boost: call_method is just a handy wrapper for PyEval_CallMethod.

Hmm, any further ideas?

4

Hmm, which version of Panda are you using? I believe on the cvs trunk we have a different way of switching threads that might be more compatible with vanilla Python calls. You could try getting the latest CVS code instead of 1.7.0.

David

5

I am using Panda 1.7. Okay, I am going to try out the CVS trunk and will report on the results.

6

Unfortunately, no difference :frowning:

7

I am working on the same problem.

I tried that. My code looks like this:

“DummyClass.h”:

#include "thread.h"

class DummyClass : public Thread
{

protected:
	virtual void thread_main();
private:
	PyObject* pyMethod;
public:
	DummyClass(PyObject* pyMethod);
	void runMe();
};

“DummyClass.cpp”

#include "DummyClass.h"

DummyClass::DummyClass(PyObject* pyMethod) : Thread("dummy","dummy")
{
	this->pyMethod = pyMethod;
}

void DummyClass::runMe()
{
	this->start(ThreadPriority::TP_normal, true);
}

void DummyClass::thread_main()
{
	this->call_python_func(this->pyMethod, NULL);
}

“DummyClassWrapper.cpp”

#include "DummyClass.h"
#include <boost/python.hpp>

BOOST_PYTHON_MODULE(DummyClass)
{
	using namespace boost::python;
	
	
	class_<DummyClass, boost::noncopyable>("DummyClass", init<PyObject*>())
		.def("run", &DummyClass::runMe);
}

“testDummyClass.py”:

import direct.directbase.DirectStart
from direct.showbase.DirectObject import DirectObject

# module import
from DummyClass import DummyClass

class EventReceiver(DirectObject):
	def __init__(self):
		DirectObject.__init__(self)
		
		self.accept('escape', self.release)
		self.d = DummyClass(self.getEvent)
		self.d.runMe()
	
	def getEvent(self):
		messenger.send('spam')
	
	def release(self):
		exit()

e = EventReceiver()   
run()

This lets the python interpreter crash when I try to create an object of that DummyClass.

What would be the way to go, if I have a C++ module, which is supposed to call a python function?

Greetings,
shr00m

8

ok, apparently I accidently linked too much libs into my code. Now I can at least create an object of my class without having the interpreter to crash.

My modified code:

“DummyClass.h” (unchanged)

#ifndef __DUMMY_CLASS_H__
#define __DUMMY_CLASS_H__

#include "thread.h"


class DummyClass : public Thread
{
protected:
	virtual void thread_main();
private:
	PyObject* pyMethod;
public:
	DummyClass(PyObject* pyMethod);
	void runMe();
};

#endif

“DummyClass.cpp”

#include "DummyClass.h"

DummyClass::DummyClass(PyObject* pyMethod) : Thread("dummy","dummy")
{
	this->pyMethod = pyMethod;
}

void DummyClass::runMe()
{
	this->start(ThreadPriority::TP_normal, true);
}

void DummyClass::thread_main()
{
	
	while(true)
	{
		//need to yield here, otherwise panda freezes (due to simple thread implementation?)
		this->consider_yield();
		// this lets everything crash
		this->call_python_func(this->pyMethod, NULL); 
	}
}

“DummyClassWrapper.cpp” (unchanged)

#include "DummyClass.h"
#include <boost/python.hpp>

BOOST_PYTHON_MODULE(DummyClass)
{
	using namespace boost::python;
	
	
	class_<DummyClass, boost::noncopyable>("DummyClass", init<PyObject*>())
		.def("runMe", &DummyClass::runMe);
}

“testDummyClass.py”

import direct.directbase.DirectStart
from direct.showbase.DirectObject import DirectObject

# module import
from DummyClass import DummyClass

class EventReceiver(DirectObject):
	def __init__(self):
		DirectObject.__init__(self)
		
		self.accept('escape', self.release)
		self.accept('s', self.start)
		self.d = DummyClass(self.getEvent)
		
	def start(self):
		self.d.runMe()

	def getEvent(self):
		messenger.send('spam')
	
	def release(self):
		exit()

e = EventReceiver()   
run()

Unfortunately it still crashes, when I call:
“call_python_func(PyObject* function, PyObject* args)”

So what is wrong with what i do? :frowning:

Greetings,
shr00m

9

You shouldn’t pass a NULL pointer as the second parameter to call_python_func(). Instead, pass a pointer to a PyTuple object–something that can safely be passed as the second parameter to PyObject_Call().

David

10

I guess I solved our weird problem by re-compiling Panda3D with threading support enabled. Now it seems to work with the snippets of my very first post.

Unless the C++ module does not call into python with an extreme high frequency, the frame rate is not affected.

We’ll see…