boost.python and Panda modules

tomkeus · April 1, 2010, 10:56am

I have certain amount of math-heavy code that I need to implement in C++ for performance reasons. I do that using boost.python to create new modules which I import to Python. Everything works well until I include panda libraries into my project.

For example, I have code which makes no reference to Panda but in my linker settings I have Panda libs as additional dependencies and that causes Python to output;

SystemError: dynamic module not initialized properly

whenever I try to import my module.

I find that very odd because I realize, that when Panda objects are referenced in my code I should call some initializations and function to make everything square with Python’s internals which in Panda case is something I have no clue about so, this is what I want to ask:

-How to use Panda’s objects in C++ and build modules with boost.python which would import properly in Python?

drwr · April 1, 2010, 5:26pm

This could be a Python version conflict. Panda is compiled to link against a particular version of Python (even if you’re not using its Python interfaces at the moment). If you link with Panda, and also link with a different version of Python, those two Python libraries will step on each other.

I can think of two solutions: (1) ensure you are using the precise same version of Python that Panda is linked with, or (2) recompile a version of Panda yourself that doesn’t link with Python at all.

David

tomkeus · April 1, 2010, 6:13pm

I am using Python that came with Panda. I have built boost.python from sources using the same Python distribution.

Let me give specific example:

For example, I have some complex math and it should return LPoint3. For example, function like this:

#include <boost/python.hpp>
#include <lpoint3.h>


LPoint3f someMath(LPoint3f& param1, ....) {
	LPoint3f result;
	
	//Do stuff
	
	return result;
}


BOOST_PYTHON_MODULE(my_module) {
	boost::python::def("someMath", someMath);
}

This will compile but module won’t work.

I know I’m missing some part of Panda API that deals with export to Python but I couldn’t find anything on that topic.

drwr · April 1, 2010, 6:54pm

Make sure you are not linking your app with pystub. This is a library that we ship with Panda that is designed to be linked into apps that do not themselves link with Python; it provides dummy stubs for all of the Python API functions that Panda has hooks into. It is necessary to avoid link errors in your non-Python program.

Since you are linking with Python, though, you should certainly not also link with pystub, and doing so will cause harm to Python.

David

tomkeus · April 1, 2010, 7:47pm

Thanks!

It was that lib3pystub.lib

Now I have troubles passing the arguments and results. Panda in Python has only Point3 but I have LPoint# for different data types.

Should I call some initializations or use some macross?

drwr · April 1, 2010, 8:17pm

The class LPoint3f in C++ is renamed to Point3 in Python.

Of course, they are not precisely the same class, because Point3 is a Python wrapper around the C++ LPoint3f. (See the Subclassing section of the manual for a more detailed explanation of this process.)

Still, it is possible to convert from one to the other without too much trouble. There are examples in the forums, as well as throughout the Panda source code. Is this what you’re asking for?

David

tomkeus · April 1, 2010, 8:24pm

Yes.

drwr · April 1, 2010, 8:48pm

#include "py_panda.h"  
extern Dtool_PyTypedObject Dtool_LPoint3f;

To convert from Point3 to LPoint3f:

PyObject *py_point = get_py_point();
LPoint3f *c_point = NULL;
DTOOL_Call_ExtractThisPointerForType(py_point, &Dtool_LPoint3f, (void **)&writer);
Py_DECREF(py_point);  // if appropriate

Note that ownership of the Python object is not passed into the C++ object, so you are still responsible for freeing (or dereferencing) the Python object after this call.

To convert from LPoint3f to Point3:

LPoint3f *c_point = new LPoint3f(get_c_point());
PyObject *py_point = DTool_CreatePyInstance((void *)c_point, Dtool_LPoint3f, true, false);

Note that ownership of the C++ object is passed into the Python object, so it should be allocated via “new” and not freed.

Also, for the record, the above operations are for LPoint3f, which does not inherit from TypedObject or ReferenceCount. For classes that do inherit from these base classes, the operations are slightly different.

David

tomkeus · April 1, 2010, 9:10pm

What is writer?

tomkeus · April 1, 2010, 9:38pm

I also get unresolved external

struct Dtool_PyTypedObject Dtool_LPoint3f" (?Dtool_LPoint3f@@3UDtool_PyTypedObject@@A)

which means I have to link some library. Which one?

tomkeus · April 1, 2010, 10:01pm

This

#include "py_panda.h"  
extern Dtool_PyTypedObject Dtool_LPoint3f;

should be (I guess)

#include "py_panda.h"  
IMPORT_THIS Dtool_PyTypedObject Dtool_LPoint3f;

With this, module builds without problem. Although I still don’t know what writer should represent.

tomkeus · April 1, 2010, 10:29pm

Silly me

writer = c_point;

Thanks a lot.

drwr · April 1, 2010, 11:16pm

Ah, bad cut-and-pasting on my part. Sorry for the confusion.

David

tomkeus · April 2, 2010, 9:37am

OK, I have little trouble here. Everything works fine until I use reference counted typed objects. Here’s simple code that demonstrates problem:

#define BOOST_PYTHON_STATIC_LIB

#include <geomVertexData.h>
#include <geomVertexFormat.h>
#include <geomVertexWriter.h>
#include <geomTristrips.h>
#include <geom.h>
#include <boost/python.hpp>
#include <py_panda.h>

IMPORT_THIS struct Dtool_PyTypedObject Dtool_Geom;

PyObject* getGeom() {
	 PT(GeomVertexData) vdata = new GeomVertexData("vertices", GeomVertexFormat::get_v3c4(), Geom::UH_static);
	 GeomVertexWriter vertex = GeomVertexWriter(vdata, "vertex");
	 GeomVertexWriter color = GeomVertexWriter(vdata, "color");
	 PT(GeomTristrips) tris = new GeomTristrips(Geom::UH_static);

	 vertex.add_data3f(-1, -1, 0); vertex.add_data3f(1, -1, 0);
	 vertex.add_data3f(1, 1, 0); vertex.add_data3f(-1, 1, 0);
	 
	 color.add_data4f(1, 0, 0, 1); color.add_data4f(1, 0, 0, 1);
	 color.add_data4f(1, 0, 0, 1); color.add_data4f(1, 0, 0, 1);
	 
	 tris->add_vertices(0, 1, 2, 3);		 
	 tris->close_primitive();
		 
	 PT(Geom) geom = new Geom(vdata);
	 geom->add_primitive(tris);
	 
	 PyObject* py_geom = NULL;
	 py_geom = DTool_CreatePyInstanceTyped((void*)geom.p(), Dtool_Geom, true, false, geom->get_type_index());
	 
	 if(py_geom != NULL)
		 return py_geom;
	 else{
		 PyErr_SetString(PyExc_RuntimeError, "Couldn't convert c_geom to py_geom");
		 return (PyObject*)-1;
	 }
}

BOOST_PYTHON_MODULE(segment)
{
	using namespace boost::python;

	def("getGeom", getGeom);
}

Executing

segment.getGeom()

in Python causes crash. What did I do wrong?

drwr · April 2, 2010, 3:59pm

The PyObject “steals” a reference to the C++ object when you pass it into DTool_CreatePyInstanceTyped(), so you have to compensate for this by explicitly calling:

geom->ref();

David

tomkeus · April 2, 2010, 4:16pm

This

#define BOOST_PYTHON_STATIC_LIB

/*
#include <lpoint3.h>
#include <lmatrix.h>
*/
#include <geomVertexData.h>
#include <geomVertexFormat.h>
#include <geomVertexWriter.h>
#include <geomTristrips.h>
#include <geom.h>
#include <boost/python.hpp>
#include <py_panda.h>


IMPORT_THIS struct Dtool_PyTypedObject Dtool_Geom;


 PyObject* getGeom() {
	 PT(GeomVertexData) vdata = new GeomVertexData("vertices", 
		 GeomVertexFormat::get_v3c4(), Geom::UH_static);

	 GeomVertexWriter vertex = GeomVertexWriter(vdata, "vertex");
	 GeomVertexWriter color = GeomVertexWriter(vdata, "color");
	 PT(GeomTristrips) tris = new GeomTristrips(Geom::UH_static);

	 vertex.add_data3f(-1, -1, 0); vertex.add_data3f(1, -1, 0);
	 vertex.add_data3f(1, 1, 0); vertex.add_data3f(-1, 1, 0);

	 color.add_data4f(1, 0, 0, 1); color.add_data4f(1, 0, 0, 1);
	 color.add_data4f(1, 0, 0, 1); color.add_data4f(1, 0, 0, 1);

	 tris->add_vertices(0, 1, 2, 3);
	 tris->close_primitive();
		 
	 PT(Geom) geom = new Geom(vdata);
	 geom->add_primitive(tris);

	 PyObject* py_geom = NULL;

	 geom->ref();
	 py_geom = DTool_CreatePyInstanceTyped((void*)geom.p(), 
		 Dtool_Geom, true, false, geom->get_type_index());

	 if(py_geom != NULL)
		 return py_geom;
	 else{
		 PyErr_SetString(PyExc_RuntimeError, "Couldn't "
			 "convert c_geom to py_geom");
		 return (PyObject*)-1;
	 }
 }


BOOST_PYTHON_MODULE(segment)
{
	using namespace boost::python;

	def("getGeom", getGeom);
}

Is is still crashing.

drwr · April 2, 2010, 5:36pm

Hmm, it works fine for me. I just pasted in your code (minus the boost stuff) into a file that’s processed by interrogate, and I can call getGeom() from Python and retrieve a perfectly valid Geom, a pair of back-to-back red triangles.

David

tomkeus · April 2, 2010, 6:59pm

Well, then it is obviously boost.python messing with the reference counting system.

drwr · April 2, 2010, 7:08pm

Can you return other kinds of PyObjects via Boost? Short of decrementing the Python reference count on your py_geom object, I’m not sure that Boost could do that much destructive here. There might be some other problem.

I’d try to narrow it down by (a) returning a standard Python object, like a PyList or something, and then (b) returning a simple Panda object like a Point3.

David

tomkeus · April 2, 2010, 7:27pm

Everything works well with such simple Panda objects. I have a function which takes Point3 parameter, converts it to LPoint3f, does some math and returns Point3 and it works without problems.