Hello all,
I have been searching for a way to replace mouse events sent to direct gui items, with touch events sent using touchpy, so i can use a multitouch screen to replace the mouse in my panda project. I am lost inside the code for direct gui and still haven’t figured out what object sends the mouse events and where they are accepted. Would anyone be so kind to point me in the right direction? Any help is much appreciated.
Thanx a lot
The mouse events are actually handled down in C++, not in Python at all. It’s handled via Panda’s concept of a “data graph”; there is a special kind of PandaNode called a MouseAndKeyboard, which generates mouse X,Y events and button events, and propagates these events down to its children.
In order to insert events from a touchscreen, then, you would either need to write a TouchScreen node (in C++) that can do the same kind of work as the MouseAndKeyboard node, or at least write a PythonMouse node (also in C++) that can be fed artificial mouse events from Python.
David
Thanx a lot for your reply. I was almost sure that it was all done in c++, the problem is that i cant find the source file/s for this. I am willing to integrate touchlib which is written in c++ with panda so I would be most grateful if you could point me to the location of those c++ files that i need to replace inside the panda source directories. Also, about your second suggestion "at least write a PythonMouse node (also in C++) that can be fed artificial mouse events from Python. " i’m not sure that i understand it. Could you elaborate more on it? Will it be an easier solution?
Look to panda/src/device/mouseAndKeyboard.h/.cxx. This is the implementation of the existing MouseAndKeyboard class, which queries the actual mouse and keyboard API’s from the graphics window and pass it through to the data graph (in the transmit_data() method).
To implement a new one of these, you will make a different class that gets its source data from some other location, but passes it to the data graph in the same way.
Making a PythonMouse class would mean making a class that gets its source data via direct user calls. In fact, let’s call it UserMouse instead of PythonMouse, since there’s nothing about it that’s particularly Python specific; it would just have methods like set_mouse_data(MouseData &md) that could be called by the user periodically to update where the “mouse” is, and it would save that data and pass it to the data graph.
I think the UserMouse class is easier to implement than a TouchScreen mouse, because integrating with a new third-party library in the C++ code is a bit of an effort. It will also be more general, because it could be used for any kind of virtual mouse that can be simulated in Python, not just for touch screens. Only downside is that it wouldn’t work without a Python layer to drive it (that is, to call setMouseData() every frame or so). But that doesn’t seem like a major limitation.
David
Thank you again for your interest in helping me. I have been looking at the code for MouseandKeyboard. From what I can tell, there is a lot of complexity in the hierarchy of classes, involving other classes such as MouseData, MouseWatcher, MouseRecorder etc. So my question is, if I just write a class that inherits from MouseandKeyboard and override the do_transmit_data method with my own code, will that be enough? It seems that then i would have to go and change all the implementation of the direct gui classes in order to work with this new class, am I right? Or maybe i could just replace the code of the original do_transmit_data with my new code?
Thanx
There are only 232 lines in mouseAndKeyboard.h and mouseAndKeyboard.cxx, put together. This is all you have to write. The other classes you mention like MouseWatcher and MouseRecorder will work properly with your class, as long as you define the proper do_transmit_data() function that sends mouse data down the pipe in the same way that MouseAndKeyboard does.
You probably don’t want to inherit from MouseAndKeyboard, since that class requires a GraphicsWindow pointer in its constructor, and your class won’t have a GraphicsWindow pointer to give it. Instead, write your own class that inherits from DataNode, the same way that MouseAndKeyboard does. You can pretty much copy MouseAndKeyboard, and just change:
(a) make the constructor receive the name of the node only,
(b) add methods to receive the current mouse data, and store it within the class, and
© change do_transmit_data() to send the stored mouse data.
David
Thank you for your advice. The MouseAndKeybord class has some attributes that need a GraphicsWindow in order to be filled and passed down the graph. For example:
int _pixel_xy_output;
int _pixel_size_output;
int _xy_output;
int _button_events_output;
int _pointer_events_output;
PT(EventStoreVec2) _pixel_xy;
PT(EventStoreVec2) _pixel_size;
PT(EventStoreVec2) _xy;
PT(ButtonEventList) _button_events;
Using my code, i can probably only fill _pixel_xy_output. Will it be all right if i dont fill the other attributes? Or maybe i can pass them down empty?
Thanx
You don’t have to fill in any of the attributes, but some objects (like the GUI!) may rely on certain of the attributes being supplied. You can always try it and see which things don’t work if you don’t supply certain attributes.
However, all of the attributes are computable by you. Nothing has anything to do specifically with a window.
The attributes are:
PT(EventStoreVec2) _pixel_xy;
This is the current (x, y) position of the mouse pointer, in “pixels”, which is to say, in the range of [0, _pixel_size_xy). Note that if the mouse pointer is < 0 or >= _pixel_size, it is considered not onscreen.
PT(EventStoreVec2) _pixel_size;
This is the maximum range of the pixel units–the value of _pixel_xy that corresponds to the lower right corner of the screen. Normally, this is the same thing as the size of the actual window, but it doesn’t have to be related to a window at all.
PT(EventStoreVec2) _xy;
This is the current (x, y) position of the mouse pointer, scaled into the range -1 … 1 in both directions.
PT(ButtonEventList) _button_events;
This is the list of button events that occurred in the previous frame. This includes mouse button events, e.g. clicks. Obviously, you don’t have to include any keyboard events in this list if you don’t have a keyboard.
David
Thank you, your advice has been very helpful so far. I am now trying to compile Panda using makepanda. I have added two files in the device directory, touchScreen.h and touchScreen.cxx. I have also added #include “touchScreen.h” in the device_composite2.cxx file and I have added the following lines in makepanda.py:
IncDirectory(“OPENCVH1”, “C:/Program Files/OpenCV/cv/include”)
IncDirectory(“OPENCVH2”, “C:/Program Files/OpenCV/cxcore/include”)
IncDirectory(“OPENCVH3”, “C:/Program Files/OpenCV/otherlibs/highgui”)
IncDirectory(“TOUCHLIB”, “C:/multitouch/touchlib/include”)
IncDirectory(“CMU”, “C:/Program Files/CMU/1394Camera/include”)
IncDirector(“VIDEOWRAPPER”, “C:/_libraries/VideoWrapper_0_1_1/include”)
IncDirectory(“TINYXML”, “C:/multitouch/touchlib/tinyxml”)
IncDirectory(“DSVL”, “C:/_libraries/dsvl-0.0.8c/src”)
LibDirectory(“OPENCV”, “C:/Program Files/OpenCV/lib”)
LibDirectory(“CMUL”, “C:/Program Files/CMU/1394Camera/lib”)
LibDirectory(“VIDEOWRAPPERL”, “C:/_libraries/VideoWrapper_0_1_1/lib”)
LibDirectory(“TINYXMLL”, “C:/multitouch/touchlib/tinyxml/lib”)
LibDirectory(“DSVLL”, “C:/_libraries/dsvl-0.0.8c/lib”)
LibName(“TOUCHLIB1”, “/panda/src/device/touchlib.lib”)
LibName(“TOUCHLIB2”, “/panda/src/device/RFFTW2dll.lib”)
Finally I have modified the OPTS variable before the TargetAdds for the device directory, to add these libraries to the compilation:
OPTS=[‘DIR:panda/src/device’, ‘BUILDING:PANDA’, ‘OPENCVH1’, ‘OPENCVH2’, ‘OPENCVH3’, ‘TOUCHLIB’, ‘CMU’, ‘VIDEOWRAPPER’, ‘TINYXML’, ‘DSVL’, ‘OPENCV’, ‘TOUCHLIB1’, ‘TOUCHLIB2’,‘CMUL’, ‘VIDEOWRAPPERL’, ‘TINYXMLL’, ‘DSVLL’]
The compilation goes well, until i get this strange error:
built/bin/interrogate -srcdir panda/src/device -Ipanda/src/device -DCPPPARSER -D__STDC__=1 -D__cplusplus -D__inline -longlong __int64 -D_X86
_ -DWIN32_VC -D_WIN32 -D_MSC_VER=1400 -D"_declspec(param)=" -D_near -D_far -D__near -D__far -D__stdcall -DFORCE_INLINING -oc built/tmp/libde
vice_igate.cxx -od built/pandac/input/libdevice.in -fnames -string -refcount -assert -python-native -Sbuilt/include/parser-inc -Ipanda/src/d
evice -S"thirdparty/win-python/include" -S"C:/Program Files/OpenCV/cv/include" -S"C:/Program Files/OpenCV/cxcore/include" -S"C:/Program File
s/OpenCV/otherlibs/highgui" -S"C:/multitouch/touchlib/include" -S"C:/Program Files/CMU/1394Camera/include" -S"C:/_libraries/VideoWrapper_0_1
_1/include" -S"C:/multitouch/touchlib/tinyxml" -S"C:/_libraries/dsvl-0.0.8c/src" -S"built/tmp" -S"built/include" -DBUILDING_PANDA -module pa
nda -library libdevice analogNode.h buttonNode.h clientAnalogDevice.h clientBase.h clientButtonDevice.h clientDevice.h clientDialDevice.h cl
ientTrackerDevice.h config_device.h device_composite.cxx dialNode.h mouseAndKeyboard.h touchScreen.h trackerData.h trackerNode.h virtualMous
e.h
*** Error in /c/multitouch/touchlib/include/Image.h near line 41, column 15:
parse error
Error parsing file: 'device_composite.cxx'There is no clue on what the error is, and from what i can tell, Image.h is fine, bacause i have compiled touchlib lots of times in the past using visual studio. Any clue on what the problem might be? Maybe I haven’t followed the right procedure in order to add these extra libraries.
The error is occurring in the interrogate step. Interrogate is a tool from Panda3D to generate python wrappers from C++ code – though it errors if it finds something nonstandard in the thirdparty header files. This happens pretty often. (Or, but less likely, it can be a bug in interrogate.)
To work around this shadow the Image.h header file in the parser-inc directory just containing the symbols you need to open up to Python, and if none, you can just leave that header file empty.
I have tried shadowing Image.h but then other problems come up, because other header files depend on it. Its a small header file:
#ifndef __TOUCHLIB_IMAGE__
#define __TOUCHLIB_IMAGE__
#include <cv.h>
#include <touchlib_platform.h>
namespace touchlib
{
// This class serves as a wrapper to OpenCV's image class
// to provide a simple interface to pixels and height/width info.
template<class T>
class Image
{
public:
IplImage* imgp;
Image(IplImage* img=0) {imgp=img;}
~Image(){imgp=0;}
int getHeight()
{
if(imgp)
return imgp->height;
else
return 0;
};
int getWidth()
{
if(imgp)
return imgp->width;
else
return 0;
};
void operator=(IplImage* img) {imgp=img;}
T* operator[](const int rowIndx)
{
return ((T *)(imgp->imageData + rowIndx*imgp->widthStep));
};
};
typedef struct
{
unsigned char b,g,r;
} RgbPixel;
typedef struct
{
float b,g,r;
} RgbPixelFloat;
typedef Image<RgbPixel> RgbImage;
typedef Image<RgbPixelFloat> RgbImageFloat;
typedef Image<unsigned char> BwImage;
typedef Image<float> BwImageFloat;
}
#endif // __TOUCHLIB_IMAGE__It seems that the problem is here: IplImage* imgp;
This doesn’t make sense because IplImage is defined somewhere inside cxtypes.h in cxcore/include, which is already in the IncDirectories.
I also noticed that interrogate creates a copy of cxtypes, containing only the definition of IplImage, in include/parser-inc:
#ifndef _CXCORE_TYPES_H_
#define _CXCORE_TYPES_H_
#include <assert.h>
#include <stdlib.h>
#include <string.h>
typedef struct IplImage
{
int nSize; /* sizeof(IplImage) */
int ID; /* version (=0)*/
int nChannels; /* Most of OpenCV functions support 1,2,3 or 4 channels */
int alphaChannel; /* ignored by OpenCV */
int depth; /* pixel depth in bits: IPL_DEPTH_8U, IPL_DEPTH_8S, IPL_DEPTH_16S,
IPL_DEPTH_32S, IPL_DEPTH_32F and IPL_DEPTH_64F are supported */
char colorModel[4]; /* ignored by OpenCV */
char channelSeq[4]; /* ditto */
int dataOrder; /* 0 - interleaved color channels, 1 - separate color channels.
cvCreateImage can only create interleaved images */
int origin; /* 0 - top-left origin,
1 - bottom-left origin (Windows bitmaps style) */
int align; /* Alignment of image rows (4 or 8).
OpenCV ignores it and uses widthStep instead */
int width; /* image width in pixels */
int height; /* image height in pixels */
struct _IplROI *roi;/* image ROI. if NULL, the whole image is selected */
struct _IplImage *maskROI; /* must be NULL */
void *imageId; /* ditto */
struct _IplTileInfo *tileInfo; /* ditto */
int imageSize; /* image data size in bytes
(==image->height*image->widthStep
in case of interleaved data)*/
char *imageData; /* pointer to aligned image data */
int widthStep; /* size of aligned image row in bytes */
int BorderMode[4]; /* ignored by OpenCV */
int BorderConst[4]; /* ditto */
char *imageDataOrigin; /* pointer to very origin of image data
(not necessarily aligned) -
needed for correct deallocation */
} IplImage;
struct IplROI
{
int coi; /* 0 - no COI (all channels are selected), 1 - 0th channel is selected ...*/
int xOffset;
int yOffset;
int width;
int height;
};
typedef struct CvRect
{
int x;
int y;
int width;
int height;
}
CvRect;
struct CvMemBlock;
struct CvMemStorage;
struct CvSeq;
struct CvSize;
struct CvArr;
//Python stub
#endifI haven’t managed to find any problem with this, except maybe for this: struct _IplROI *roi;
Still, the thing is that i dont even want to create python code for anything more than my class touchScreen.h, but one class depends on another, and if I shadow something using the parser-inc directory, then I end up with an error
IplImage* imgp;That isn’t a redefinition of IplImage at all – it just says that imgp contains a pointer to an IplImage instance. So that should be right.
The cxtypes.h wasn’t auto-generated at all – Panda3D already supports OpenCV for some time (though not through makepanda yet, but I plan on adding it for 1.6.0), and thus this file was needed. When that file was added, though, the OpenCV version used was old – so this header file might not reflect newer versions of OpenCV anymore. You can just go ahead and update that file to the latest version of OpenCV.
Something’s not right. By “shadow with the parser-inc directory”, we mean to put the shadow file within that directory, which is not searched by the C++ compiler! The parser-inc directory is only searched by interrogate, not the actual C++ compiler, so putting any files within parser-inc should have no effect whatsoever on the actual compilation of your code. It just serves to help interrogate know the typenames that are necessary in order to parse your code.
Interrogate doesn’t actually compile the code, it just needs to be able to read and understand your part of the code so it can generate interfaces. This means it needs to know which symbols are type names and which symbols aren’t (this is necessary in order to parse C/C++ code), but other than that it doesn’t really need to parse any header file that it’s not generating interfaces for.
So, usually, a blank file in parser-inc, or at most one containing a couple of trivial typedef commands, solves the problem.
David
I know that it’s not a redefinition, that’s why I have been busting my head to figure out why interrogate always finds a parse error on that line, no matter where I move this line, the parse error is always there. Something else which is interesting is that if I delete this line then interrogate crashes.
Actually that’s exactly what I have done, at least by including the cxcore/include directory I thought that the new cxtypes.h header would be considered instead of the one in parser-inc. Actually I even tried replacing the cxtypes.h header in parser-inc with the one found in cxcore/include (in OpenCV) but every time i run makepanda and the interrogate step comes up, I realise that the file that I replace in parser-inc is overwritten to its previous form. That is why I mentioned that it is auto-generated, in fact I have no idea why it keeps overwriting my file.
I understood what pro-rsoft meant with shadowing. I probably haven’t made my self clear by using the word ‘error’. By that I meant that I get an error in interrogate, not in compilation. So, if I place an empty Image.h in parser-inc, then the parse error moves to an other header file, if I place an empty copy of that header in parser-inc, then the parse error moves to another header file and so on, until after I do this for 3-4 headers, then the parse error comes up in my class touchScreen.h where I can’t replace or delete any code because I need interrogate to produce a wrapper for it (from what I have understood so far).
I apologise for my long posts and my inability to resolve this despite your help. I hope I have explained myself better this time.
Thanx a lot for your time in helping me out.
If the error occurs within your own class, it’s probably something nonstandard in your code (or a bug in interrogate). Can you post the erroring lines in your code?
Or, if the error comes up on a line that uses a type that should have been defined in Image.h or one of the other header files that you stubbed out, then it just means you need to make a trivial typedef for that type in your stub header files. Something like:
This is necessary because the nature of C syntax requires the parser to know whether a particular symbol is a type or an instance name, before it can even parse it. But beyond that, it doesn’t need to know any details of the type, so if you lie and say it is a class or struct, or even a typedef of an int, it should be just fine.
David
The error is exactly what drwr mentioned. So I now have an idea of what I need to do… I really hope this works out. Unfortunately I’m left without my pc for a few days, so unless I can get my hands on another pc and repeat the process earlier, I will probably test this on Monday. Thanx a lot for your help drwr and pro-rsoft.
I finally managed to compile Panda with my new class and 3rd party libraries. I was happy to see the Touchlib library code being run once i create an object of my class and attach it to the render node. The problem now is, how do I transmit the events down the graph? In the MouseAndKeyboard class comments I noticed that I need to attach an EventThrower object to the object of my class, otherwise the events will be discarded. I couldn’t find such a class in the Panda Reference. Any hints on how to do that?
Panda’s “data graph” is designed to transmit user-input type data from producer nodes (like the MouseAndKeyboard) to consumer nodes (like the MouseWatcher or ButtonThrower). The data is automatically transmitted along the scene graph connections, from parent node to all its child nodes.
It is the function do_transmit_data() which effects this operation for a particular node. Panda will call this function for each node in the data graph–that is, for each node parented directly or indirectly to base.dataRoot. The inputs supplied to do_transmit_data() are the events that are received from the node’s parents, and the outputs that do_transmit_data() returns are supplied in turn to the node’s children.
So, if you have written do_transmit_data() correctly in your new class, such that it makes the “mouse events” available in the outputs list, then all you need to do is attach your node to the data graph, and attach the appropriate receiving nodes to your node. One such node is the ButtonThrower node (formerly called the EventThrower). Normally, simply parenting a ButtonThrower to the mouse node would be sufficient to cause Panda events to be generated in response to mouse button events.
If you intend your touch interface to replace the mouse for GUI, though, you will need to attach the MouseWatcher to your touch interface node. This is the node that drives the GUI interfaces. When Panda creates a MouseAndKeyboard node, it normally attaches the MouseWatcher to the MouseAndKeyboard, then attacks a ButtonThrower to that. In fact, if your goal is to completely replace the mouse, you should simply reparent the MouseWatcher node to your own node, something like this:
np = base.dataRoot.attachNewNode(MyTouchNode('touch'))
base.mouseWatcher.reparentTo(np)If that doesn’t seem to work, there might be a problem in your do_transmit_data() function. You can help diagnose problems by turning on spam reporting for the actual data transmission. Put this in your Config.prc file:
notify-level-dgraph spamand it will tell you in excruciating detail about each data element as it travels from node to node in the data graph.
David
Your reply has been very helpful. I followed your advice and it seems that my own do_transmit_data method gets called when I reparent the mouseWatcher node to my touch node(i wrote a couple of cout inside it for debugging). However I was sad to see that python crashes immediately afterwards 
In other words, my debug message gets printed and then I get a message Python.exe has encountered a problem and needs to close etc.
My code for do_transmit_data is:
void TouchScreen::
do_transmit_data(DataGraphTraverser *, const DataNodeTransmit &,
DataNodeTransmit &output) {
//Screen is a private field of the class
screen->getEvents();
std::cout << "testing"<< std::endl;
FingerMap::iterator iter;
//Get the first blob from the blobs map (just for testing)
iter=fingermap.begin();
int id = (*iter).first;
TouchData data = (*iter).second;
_pixel_xy->set_value(LPoint2f(data.X, data.Y));
output.set_data(_pixel_xy_output, EventParameter(_pixel_xy));
//I replaced window size with screen size because I don't have a window to pass it
_pixel_size->set_value(LPoint2f(1280, 1024));
output.set_data(_pixel_size_output, EventParameter(_pixel_size));
// Normalize pixel motion to range [-1,1].
float xf = (float)(2 * data.X) / (float)1280 - 1.0f;
float yf = 1.0f - (float)(2 * data.Y) / (float)1024;
_xy->set_value(LPoint2f(xf, yf));
output.set_data(_xy_output, EventParameter(_xy));
std::cout << "testing 2"<< std::endl;
}and I noticed that testing gets printed while testing 2 doesn’t get printed at all.
Also, when my program runs I get a warning:
:interrogatedb(warning): Class TouchScreen has a zero TypeHandle value; check that init_type() is called.
and before ‘testing’ is printed I get a warning:
Type TouchScreen was unregistered!
Could that be the problem, because somehow init_type() is not called?
Any ideas?
Thanx a lot